layout engine, text, content model, etc
Erik van der Poel
works with text (content model, frame model, etc). I'm not sure that I
got it right, so could you please correct me if I'm wrong? Thanks!
Jonathan Rosenne wrote:
>
> There should be no interaction between the bidi algorithm and the HTML tags.
> The bidi algorithm should be applied on a block of text after resolution of
> the HTML source, including NCRs and tags that resolve to bidi formatting
> codes, and the character attributes such as color and font applied according
> to the resulting reordering.
What exactly do you mean when you say "block of text"? Do you mean a
whole paragraph? Or a line that has an explicit <BR>? Or an inline
element such as <I>...</I>?
I'm not sure whether it would be easy to access a whole paragraph of
text for the purposes of applying the bidi algorithm. I haven't looked
very closely at the architecture of the layout engine recently, but my
understanding is as follows.
The HTML document enters the system via the networking library and the
parser. As the parser parses the document, the "content model" (or DOM,
Document Object Model) is built up. It's a tree.
The layout engine then applies style sheets to the content to build up a
"frame model", which corresponds quite closely to CSS's line and inline
boxes. For example, a long paragraph is split into multiple lines by
measuring the text and finding appropriate line break points. Inline
elements such as <B>, <I> and <FONT> also receive at least one frame
each, more than one if the text has to be split across multiple lines.
The font engine itself does not get to see very much text at a time. The
DrawString method will see at most one line of text. The GetWidth method
may see more than that when the layout engine is trying to determine
line break points. Both of these methods see less than a line of text
when a short inline element is being measured or drawn (e.g.
<B>...</B>).
As far as I know, the content model stores the text in a hierarchical
data structure. There is a node for each paragraph. There could be more
than one node under a paragraph node if that paragraph has inline
elements such as <A HREF="...">...</A>. I believe that the text in a
paragraph is not necessarily stored in a single long buffer.
If the bidi algorithm must be applied on a whole paragraph (I believe it
must), then we could access the text of the paragraph by "traversing"
(climbing up and down) the content model. I don't know how difficult or
error-prone this would be.
Alternatively, we could lobby for a change to the content model to make
a paragraph's text be stored in a single buffer, with the content
model's nodes pointing into various points in the buffer (offsets). I
don't know if this would really make it easier to apply the bidi
algorithm and to build up the frame model. It might make the bidi
algorithm easy, but the frame model too hard.
It might be a good idea to try both approaches (i.e. single text buffer
for block elements vs multiple buffers for inline elements) in the
content model to see which one is "best" (ease of programming, simple to
understand, efficient in terms of memory and speed, etc).
Erik
Jonathan Rosenne
> -----Original Message-----
> From: Erik van der Poel [mailto:er...@netscape.com]
> Sent: Sunday, December 12, 1999 12:35 AM
> To: Jonathan Rosenne
> Cc: ft...@netscape.com; mozill...@mozilla.org; Tzafrir Cohen;
> mozilla...@mozilla.org
> Subject: layout engine, text, content model, etc
>
>
> mozilla-layout people: Below, I try to explain how the layout engine
> works with text (content model, frame model, etc). I'm not sure that I
> got it right, so could you please correct me if I'm wrong? Thanks!
>
> Jonathan Rosenne wrote:
> >
> > There should be no interaction between the bidi algorithm and
> the HTML tags.
> > The bidi algorithm should be applied on a block of text after
> resolution of
> > the HTML source, including NCRs and tags that resolve to bidi formatting
> > codes, and the character attributes such as color and font
> applied according
> > to the resulting reordering.
>
> What exactly do you mean when you say "block of text"? Do you mean a
> whole paragraph? Or a line that has an explicit <BR>? Or an inline
> element such as <I>...</I>?
HTML 4 defines block elements and in-line elements. P and BR are bloc, I is
inline.
>
> I'm not sure whether it would be easy to access a whole paragraph of
> text for the purposes of applying the bidi algorithm. I haven't looked
> very closely at the architecture of the layout engine recently, but my
> understanding is as follows.
It may not be easy, but this is the spec.
>
> The HTML document enters the system via the networking library and the
> parser. As the parser parses the document, the "content model" (or DOM,
> Document Object Model) is built up. It's a tree.
>
> The layout engine then applies style sheets to the content to build up a
> "frame model", which corresponds quite closely to CSS's line and inline
> boxes. For example, a long paragraph is split into multiple lines by
> measuring the text and finding appropriate line break points. Inline
> elements such as <B>, <I> and <FONT> also receive at least one frame
> each, more than one if the text has to be split across multiple lines.
The bid algorithm should be performed before the text is split into lines.
>
> The font engine itself does not get to see very much text at a time. The
> DrawString method will see at most one line of text. The GetWidth method
> may see more than that when the layout engine is trying to determine
> line break points. Both of these methods see less than a line of text
> when a short inline element is being measured or drawn (e.g.
> <B>...</B>).
By the time you reach fonts, you should be past bidi reordering.
Roozbeh Pournader
On Sun, 12 Dec 1999, Jonathan Rosenne wrote:
> > [...] The GetWidth method
> > may see more than that when the layout engine is trying to determine
> > line break points. Both of these methods see less than a line of text
> > when a short inline element is being measured or drawn (e.g.
> > <B>...</B>).
> By the time you reach fonts, you should be past bidi reordering.
You are some way right and some way wrong. Bidi algorithm has two parts.
One before line breaking and one after that. One of the reasons that we
should start BIDI before line breaking, is the character width of mirrored
characters (parentheses in RTL context should be displayed mirrored, left
parenthesis becoming right parenthesis.) I don't know how will this be
implemented, but the easiest approach is to trade the left parenthesis for
a right one. But the character width of these may be different. So
GetWidth should know which one will get rendered. I don't know what will
happen with kerning when mirroring, but that may also make problems.
Apart from the possible change in width, I can see no other part in
Unicode 3.0 BIDI algorithm that can't be done after line breaking. If you
have the complete text of the paragraph, together with knowledge about
inherited "dir" attributes, there is no need to be there before line
breaking.
Do you know any other cases?
--Roozbeh
Jonathan Rosenne
than one reason?
Additionally, HTML 4 and Unicode specify that the unit for reordering is a
block, such as a paragraph.
Jony
> -----Original Message-----
> From: Roozbeh Pournader [mailto:roo...@sina.sharif.ac.ir]
> Sent: Sunday, December 12, 1999 3:53 AM
> To: Jonathan Rosenne
> Cc: Erik van der Poel; ft...@netscape.com;
> mozill...@mozilla.org; Tzafrir Cohen; mozilla...@mozilla.org
Erik van der Poel
>
> The bidi algorithm should be performed before the text is split into
> lines.
I found the latest bidi algorithm spec at:
http://www.unicode.org/unicode/reports/tr9/
It says that the bidi algorithm proceeds in 3 main phases:
Separation of the input text into paragraphs. The rest of the
algorithm affects only the text between paragraph separators.
Resolution of the embedding levels of the text. In this phase,
the directional character types, plus the explicit format
codes, are used to produce resolved embedding levels.
Reordering the text for display on a line-by-line basis using
the resolved embedding levels, once the text has been broken
into lines.
So, I think the 3rd phase should be performed *after* the text is split
into lines.
The 1st phase clearly occurs when we parse the document and create the
content model.
The 2nd phase could be performed as we create the frame model from the
content model and the style sheets, storing the resolved embedding
levels in the frame model.
Alternatively, we could store the resolved embedding levels in the
content model, but we need to be careful not to emit that info when we
are the editor, writing the document out to a file.
If we store the resolved embedding levels in the content model, we may
be able to expose that info via DOM APIs. If we store it in the frame
model, we may be able to expose it via CSSOM APIs. Has anyone in the W3C
working groups expressed a need to expose the resolved embedding level
info via standardized APIs?
Now let's come back to the reordering again. If you look at the text as
it moves through the system, you notice that it gets copied several
times. Initially, the document arrives in some charset (character
encoding) from the Net or a file. The bytes are stored in memory
somewhere. Then we convert these bytes into 16-bit Unicodes, copying the
text to a different buffer. Finally, on some platforms we must convert
the Unicodes to the font API's encoding, thereby copying the text to yet
another buffer.
I think it would be wasteful if we had to create yet another buffer to
reorder the text, so how about performing the reordering at the same
time that we are converting from Unicode to the font API's encoding?
I think it makes sense to bring the reordering closer to the fonts,
because the bidi algorithm spec also says:
When working with bidirectional text, the characters are still
interpreted in logical order--only the display is affected.
Under my proposal, we would end up with text in 3 different kinds of
buffers:
(1) input buffer: text in document's charset, in logical order
(2) processing buffer: text in Unicode, in logical order
(3) output buffer: text in font API's encoding, in visual order
(Of course, in the editor's case, (1) would be an output buffer.)
Erik
Yaacov Akiba Slama
As Mike Kaply wrote, I am working for IBM and I am the team leader for the
Israeli half of the bidi team. The other half is in Cairo and leaded by Maha
Abou El-Rous.
We'll post some detailled plans of our work later, but I wanted to add some
technical comments (they are based on what I saw in this newsgroup, on what
I understand in the Mozilla code, and on our past experiences with Netscape
4.x [in which we implemented bidi in the OS/2, AIX and Windows]).
1) All the bidi-algoritms needs to be implemented in the cross platform part
of Mozilla, and has to work with every platform in every language as soon as
this platform has fonts containing the right glyphs.
2) The resolving of the levels needs to be performed in the layout engine.
3) The content model will contain the logical text (un-reordered), and the
level info.
4) The frame model will contain the visual text (after reodering AND
shaping - ie the shaping code will be here, and the platform dependant code
will perhaps have to do some conversion [unicode<->codepage etc..] but no
shaping). Having the visual representation of the text here helps a lot when
dealing with selection and cursor positionning.
5) We need to ensure that no reordering/shaping is done by the platform.
I wanted to be more specific about this last point in Windows.
a) On non bidi windows (95,98 and NT), the solution is trivial since no
reordering/shaping is done by the platform.
b) On all Windows NT, Windows 2000, the best is to call
GetCharacterPlacement (GCP) to obtain glyphs, and then ExTextOut (ETO) with
ETO_GLYPHS flag. This si OK because these paltforms support the Unicode
version of GCP.
GCP can also be used for getWidth() because it returns the same info as
GetTextExtentPoint32() (GTE).
c) On all version of Win, we can also detect if the platform support
reordering/shaping and in this case, insert LRM (Left Right Marker which is
a left to right character of [theoretically] zero width) before/after each
strong right to left char. After that, we can call both ETO and GTE. This is
the solution we used in NS 4.6.
The problem is taht there are inconsistancies between Windows versions (for
instance in 95, only the Ansi/8bits versions of the functions does
reordering/shaping, in 98, both Ansi and Unicode versions does
reordering/shaping etc..)
d) The last solution is to use the code already present which reads the cmap
table of the (TrueType) font and obtain the glyph corresponding to each
(Unicode) character. So we can use it to call ETO. The problem here is that
there is no glyph version of GTE, so we don't know how to obtain the width.
My solution (if someone has a better one, that would be great) is to use ETO
with a DC configured by SetTextAlign with the TA_UPDATECP. In this case ETO
modify the CP (current position) in the DC, and by querying this pos, we can
have the width of the text. Of course we needs to have a dummy DC, if we
only want the width.
Yaacov Akiba Slama
Erik van der Poel <er...@netscape.com> wrote in message
news:38534BBF...@netscape.com...
Roozbeh Pournader
On Sat, 11 Dec 1999, Erik van der Poel wrote:
> I found the latest bidi algorithm spec at:
> Resolution of the embedding levels of the text. In this phase,
> the directional character types, plus the explicit format
> codes, are used to produce resolved embedding levels.
>
> Reordering the text for display on a line-by-line basis using
> the resolved embedding levels, once the text has been broken
> into lines.
As I told in the previous message, before breaking, embedding levels will
be used to find the mirroring characters. That may affect line breaking.
> So, I think the 3rd phase should be performed *after* the text is split
> into lines.
It must.
> I think it would be wasteful if we had to create yet another buffer to
> reorder the text, so how about performing the reordering at the same
> time that we are converting from Unicode to the font API's encoding?
>
> I think it makes sense to bring the reordering closer to the fonts,
> because the bidi algorithm spec also says:
>
> When working with bidirectional text, the characters are still
> interpreted in logical order--only the display is affected.
I agree with this. Consider a user trying to search. The browser should
have the Unicode version available. Consider that there is a
"NETSCAPE 5.0" in text (upper letters are considered RTL), which
get out "5.0 EPACSTEN". Then the users searchs for "NETSCAPE 5".
If we perform bidi on that and give the BIDI thing to the searcher,
it will not find "5 EPACSTEN".
--Roozbeh
Roozbeh Pournader
On Sun, 12 Dec 1999, Jonathan Rosenne wrote:
> Kerning is very important in this context and is required. Do you need more
> than one reason?
I do not know much about the kerning mechanism of RTL-enabled fonts. It is
possible that they have considered the case, so we can break the line even
not knowing about mirroring the parenthesis, and the behaviour being the
same.
> Additionally, HTML 4 and Unicode specify that the unit for reordering is a
> block, such as a paragraph.
Of course, I have read them carefully. But I think that mirroring is the
only problem, and there is a possiblity that it can be addressed some
other way.
--Roozbeh
Erik van der Poel
>
> Hello All!
Hello, Yaacov. Thanks for forming the new bidi team!
> our past experiences with Netscape
> 4.x [in which we implemented bidi in the OS/2, AIX and Windows]).
Did you use XOM on AIX? Or lowest-level Xlib (XDrawString*, etc)?
> 3) The content model will contain the logical text (un-reordered), and the
> level info.
As I mentioned in my previous message, that is one of the possibilities.
Could you tell me why you would want to store the level info in the
content model instead of the frame model?
> 4) The frame model will contain the visual text (after reodering AND
> shaping - ie the shaping code will be here, and the platform dependant code
> will perhaps have to do some conversion [unicode<->codepage etc..] but no
> shaping).
Currently, the text is stored in 3 kinds of buffers:
(1) input buffer: cached; text in doc's charset
(2) processing buffer: cached; text in Unicode
(3) output buffer: not cached; text in font API's encoding
I identified the inefficiency of (3) a long time ago, and raised the
issue with the layout team, who preferred not to cache the output buffer
for now. It seemed like they were willing to entertain that thought
later, after getting a robust first version of the product out the door.
Now you are proposing the following architecture:
(a) input buffer: cached; text in doc's charset; logical order
(b) processing buffer: cached; text in Unicode; logical order
(c) pre-output buffer: cached; text in Unicode; reordered and shaped
(d) output buffer: not cached; font API's encoding; reordered/shaped
This seems like the worst of both worlds: We are using more memory for
(c), *and* we are still wasting time in (d). Maybe we should merge (c)
and (d), and either cache it, or not.
Given that the current architecture does not cache the output buffer,
how about keeping this architecture for the initial experimental
implementation of bidi, and then evaluating the results before deciding
to cache the output buffer?
> Having the visual representation of the text here helps a lot when
> dealing with selection and cursor positionning.
I realize that selection and caret positioning are complicated in bidi,
but have you tried selecting some text in a document using HTML DIR
attributes and/or CSS direction properties in the current Mozilla? I
tried it a while ago, and was surprised to find that Mozilla already
supports discontiguous selections (the highlighting is displayed in more
than one rectangle).
> 5) We need to ensure that no reordering/shaping is done by the platform.
I agree that it would be cleaner if we could do the reordering and
shaping in all cases, but if it turns out to be very difficult to
circumvent the platform's reordering and/or shaping on some platforms,
why not let those platforms do it?
> b) On all Windows NT, Windows 2000, the best is to call
> GetCharacterPlacement (GCP) to obtain glyphs, and then ExTextOut (ETO) with
> ETO_GLYPHS flag. This si OK because these paltforms support the Unicode
> version of GCP.
> GCP can also be used for getWidth() because it returns the same info as
> GetTextExtentPoint32() (GTE).
We should measure the performance of GCP on non-bidi text before
deciding to use it for such text. I heard that it is slower than GTE.
One of the most sensitive areas in Web browser performance is text
measurement.
> c) On all version of Win, we can also detect if the platform support
> reordering/shaping and in this case, insert LRM (Left Right Marker which is
> a left to right character of [theoretically] zero width) before/after each
> strong right to left char.
Alternatively, you can split the text into pieces which are each either
LTR or RTL, right? I thought that the bidi algorithm would not only
determine embedding levels, but also help us split up the text in this
way.
> The problem is taht there are inconsistancies between Windows versions (for
> instance in 95, only the Ansi/8bits versions of the functions does
> reordering/shaping, in 98, both Ansi and Unicode versions does
> reordering/shaping etc..)
It is possible to detect Win95 vs 98, right?
> d) The last solution is to use the code already present which reads the cmap
> table of the (TrueType) font and obtain the glyph corresponding to each
> (Unicode) character. So we can use it to call ETO. The problem here is that
> there is no glyph version of GTE, so we don't know how to obtain the width.
> My solution (if someone has a better one, that would be great) is to use ETO
> with a DC configured by SetTextAlign with the TA_UPDATECP. In this case ETO
> modify the CP (current position) in the DC, and by querying this pos, we can
> have the width of the text. Of course we needs to have a dummy DC, if we
> only want the width.
This would mean an in-memory DC, right? So the ETO routine would
actually copy the glyph bitmaps into the DC's memory. This seems slow,
and as I mentioned, text measurement speed is critical. It might be
faster if we do the measurement ourselves, based on the font's metrics.
That code might be harder to write (kerning, etc). Here again, I wonder
if it would be better to allow the platform to reorder/shape the text,
instead of bending over backwards trying to do it ourselves.
Erik
Erik van der Poel
>
> As I told in the previous message, before breaking, embedding levels will
> be used to find the mirroring characters. That may affect line breaking.
To measure text, we have to convert from Unicode to the font API's
encoding anyway, so how about performing the mirroring inside our text
measurement routine? I.e. we have the logical characters in the
processing buffer, and we have mirrored characters in the output buffer.
Erik
Roozbeh Pournader
Yes, this solves the thing, but that was before your purposal.
You were telling about performing the BIDI after line breaking.
Roozbeh
Jonathan Rosenne
so it should allow user friendly input and editing of bidi text. In this
case reordering should be performed on the fly.
Jony
Jonathan Rosenne
http://www.qsm.co.il/Software/Measure.htm
Jony
> -----Original Message-----
> From: Erik van der Poel [mailto:er...@netscape.com]
Yaacov Akiba Slama
Erik van der Poel <er...@netscape.com> wrote in message
news:3853E9BF...@netscape.com...
> Yaacov Akiba Slama wrote:
> >
> > Hello All!
>
> Hello, Yaacov. Thanks for forming the new bidi team!
>
> > our past experiences with Netscape
> > 4.x [in which we implemented bidi in the OS/2, AIX and Windows]).
>
> Did you use XOM on AIX? Or lowest-level Xlib (XDrawString*, etc)?
All the reordering/shaping is done in the BE of Netscape.
OK to merge (c) and (d) [assuming the reordering/shaping is done in Unicode
before the conversion] but if (d) is not cached, the biggest problem are
with Arabic text, because the number of glyphs after shaping can be
different (more or less) than the number of chars in the original string.
But we can handle that.
> > Having the visual representation of the text here helps a lot when
> > dealing with selection and cursor positionning.
>
> I realize that selection and caret positioning are complicated in bidi,
> but have you tried selecting some text in a document using HTML DIR
> attributes and/or CSS direction properties in the current Mozilla? I
> tried it a while ago, and was surprised to find that Mozilla already
> supports discontiguous selections (the highlighting is displayed in more
> than one rectangle).
>
I tried to create an HTML with the dir attribute, but I cannot see any
effect in Mozilla. Do you hav an example ?
Concerning the discontiguous selection, this will help us.
> > 5) We need to ensure that no reordering/shaping is done by the platform.
>
> I agree that it would be cleaner if we could do the reordering and
> shaping in all cases, but if it turns out to be very difficult to
> circumvent the platform's reordering and/or shaping on some platforms,
> why not let those platforms do it?
>
One reason is that the reordering algorithm of Windows 95/98 is different
than the Unicode one.
> > b) On all Windows NT, Windows 2000, the best is to call
> > GetCharacterPlacement (GCP) to obtain glyphs, and then ExTextOut (ETO)
with
> > ETO_GLYPHS flag. This si OK because these paltforms support the Unicode
> > version of GCP.
> > GCP can also be used for getWidth() because it returns the same info as
> > GetTextExtentPoint32() (GTE).
>
> We should measure the performance of GCP on non-bidi text before
> deciding to use it for such text. I heard that it is slower than GTE.
> One of the most sensitive areas in Web browser performance is text
> measurement.
>
> > c) On all version of Win, we can also detect if the platform support
> > reordering/shaping and in this case, insert LRM (Left Right Marker which
is
> > a left to right character of [theoretically] zero width) before/after
each
> > strong right to left char.
>
> Alternatively, you can split the text into pieces which are each either
> LTR or RTL, right? I thought that the bidi algorithm would not only
> determine embedding levels, but also help us split up the text in this
> way.
>
If you mean that we can call the platform functions (GTE, ETO) by runs of
only strong RTL chars (after returning the string) or only non strong RTL
chars, you are true, but there is a performance hit.
Concerning the bidi algorithm, even if it says that a run of chars is LTR
(for instance) doesn't mean that no chars in this run is strong RTL. A
simple example is when you use LRO (left to right override) in the very
beginning of the string (which the platform function doesn't see and even if
it see it, Win 95/98 doesn't understand it).
> > The problem is taht there are inconsistancies between Windows versions
(for
> > instance in 95, only the Ansi/8bits versions of the functions does
> > reordering/shaping, in 98, both Ansi and Unicode versions does
> > reordering/shaping etc..)
>
> It is possible to detect Win95 vs 98, right?
True, but we need to test each version of Win (95, 98, 98 SE, Millenium) and
even in one platform, there are things that depends on the font (for
instance the width of LRM is not the same in David[1] and TimesNewRoman[0]).
>
> > d) The last solution is to use the code already present which reads the
cmap
> > table of the (TrueType) font and obtain the glyph corresponding to each
> > (Unicode) character. So we can use it to call ETO. The problem here is
that
> > there is no glyph version of GTE, so we don't know how to obtain the
width.
> > My solution (if someone has a better one, that would be great) is to use
ETO
> > with a DC configured by SetTextAlign with the TA_UPDATECP. In this case
ETO
> > modify the CP (current position) in the DC, and by querying this pos, we
can
> > have the width of the text. Of course we needs to have a dummy DC, if we
> > only want the width.
>
> This would mean an in-memory DC, right? So the ETO routine would
> actually copy the glyph bitmaps into the DC's memory. This seems slow,
> and as I mentioned, text measurement speed is critical. It might be
> faster if we do the measurement ourselves, based on the font's metrics.
> That code might be harder to write (kerning, etc). Here again, I wonder
> if it would be better to allow the platform to reorder/shape the text,
> instead of bending over backwards trying to do it ourselves.
>
We need perhaps to do some tests, because the TrueType renderer has some
cache mechanism.
We can also try to see how complicated is to do the measurement ourself by
using the metrics. Except the kerning (which is in a simple table of the
font), what can restrain us to do simples additions ?
> Erik
>
Yaacov Akiba
Erik van der Poel
>
> I tried to create an HTML with the dir attribute, but I cannot see any
> effect in Mozilla. Do you hav an example ?
I just tried it with the following:
<p dir="rtl">One <b>two</b> <i>three</i> four.</p>
It has some interesting selection behavior.
> > I agree that it would be cleaner if we could do the reordering and
> > shaping in all cases, but if it turns out to be very difficult to
> > circumvent the platform's reordering and/or shaping on some platforms,
> > why not let those platforms do it?
>
> One reason is that the reordering algorithm of Windows 95/98 is different
> than the Unicode one.
Ouch. Well, if it's not too difficult to bypass the platform's
reordering, then let's do so, but if it *is* very difficult, maybe our
initial version should just use the Windows algorithm. Does IE implement
the Unicode bidi spec correctly, even on the various Windows versions?
> We can also try to see how complicated is to do the measurement ourself by
> using the metrics. Except the kerning (which is in a simple table of the
> font), what can restrain us to do simples additions ?
Kerning is one thing that we would need to keep in mind. Another thing
is speed. Text measurement is critical. Also, we would need to think
about the units we use internally. If we compute measurements in pixels,
the numbers may be rounded too much. The TrueType font may have better
(larger) numbers to work with, in terms of "units per em" (often 2048).
Erik
Erik van der Poel
>
>
> <p dir="rtl">One <b>two</b> <i>three</i> four.</p>
By the way, when I say "Mozilla", I mean the new open source version,
not the old Netscape Communicator 4.X version.
Also, the fact that <html dir="rtl"> does not work is a known bug:
http://bugzilla.mozilla.org/show_bug.cgi?id=19312
Erik
Erik van der Poel
The mail composer and document composer use Mozilla's editor, which uses
DOM APIs to edit the content model. Whenever the content model changes
(e.g. editing, packets arriving from network, etc), the layout engine
updates the appropriate parts of the frame model(s) automatically. This
is called incremental reflow.
So, if we implement bidi correctly in the content and/or frame models,
the reordering should also be performed correctly when the user enters
or edits text.
Erik
Jonathan Rosenne
you shouldn't worry too much about it. If the platform does not allow you to
bypass reordering then it cannot support any bidi word processing software.
These problems are not unique to a browser.
Jony
> -----Original Message-----
> From: Erik van der Poel [mailto:er...@netscape.com]
> Sent: Monday, December 13, 1999 9:55 PM
> To: Yaacov Akiba Slama
> Cc: mozill...@mozilla.org; mozilla...@mozilla.org
> Subject: Re: layout engine, text, content model, etc
>
>
> Yaacov Akiba Slama wrote:
> >
> > I tried to create an HTML with the dir attribute, but I cannot see any
> > effect in Mozilla. Do you hav an example ?
>
> I just tried it with the following:
>
> <p dir="rtl">One <b>two</b> <i>three</i> four.</p>
>
Jonathan Rosenne
for user friendly bidi editing this is not enough. A naive implementation
would cause the text to jump about wildly during inserts and deletes. One
needs to make some assumptions about the properties of the next character to
be input and handle with good grace the case that the guess is wrong.
Jony
> -----Original Message-----
> From: Erik van der Poel [mailto:er...@netscape.com]
> Sent: Monday, December 13, 1999 11:32 PM
> To: Jonathan Rosenne
> Cc: Yaacov Akiba Slama; mozilla...@mozilla.org;
> mozill...@mozilla.org
> Subject: Re: layout engine, text, content model, etc
>
>
Frank Yung-Fong Tang
Jonathan Rosenne wrote:
> Erik,
>
> you shouldn't worry too much about it. If the platform does not allow you to
> bypass reordering then it cannot support any bidi word processing software.
> These problems are not unique to a browser.
>
> Jony
Mac support Arabic/Hebrew word process very well.
Tell me how to bypass bi-di reordring on Mac ? I don't think you can.
Frank Yung-Fong Tang
Yaacov Akiba Slama wrote:
> Hello All!
>
> As Mike Kaply wrote, I am working for IBM and I am the team leader for the
> Israeli half of the bidi team. The other half is in Cairo and leaded by Maha
> Abou El-Rous.
>
> We'll post some detailled plans of our work later, but I wanted to add some
> technical comments (they are based on what I saw in this newsgroup, on what
> I understand in the Mozilla code, and on our past experiences with Netscape
> 4.x [in which we implemented bidi in the OS/2, AIX and Windows]).
>
> 1) All the bidi-algoritms needs to be implemented in the cross platform part
> of Mozilla, and has to work with every platform in every language as soon as
> this platform has fonts containing the right glyphs.
Great..
> 2) The resolving of the levels needs to be performed in the layout engine.
Agree
> 3) The content model will contain the logical text (un-reordered), and the
> level info.
Agree.
> 4) The frame model will contain the visual text (after reodering AND
> shaping - ie the shaping code will be here, and the platform dependant code
> will perhaps have to do some conversion [unicode<->codepage etc..] but no
> shaping). Having the visual representation of the text here helps a lot when
> dealing with selection and cursor positionning.
How about Mac ? I don't think you can give visual / shaped Unicode to MacOS and
expect it render ti correctly for you. On Mac, it expect unshapped logical order
as input for rendering.
>
>
> 5) We need to ensure that no reordering/shaping is done by the platform.
I don't think you have a way to do that on the Mac. I think it is easy to do
that on X, but I don't think you can find a way to by-pass reordering/shaping
code on Mac.
Yaacov Akiba Slama
Erik van der Poel <er...@netscape.com> wrote in message
news:38554F23...@netscape.com...
> Yaacov Akiba Slama wrote:
> >
> > I tried to create an HTML with the dir attribute, but I cannot see any
> > effect in Mozilla. Do you hav an example ?
>
> I just tried it with the following:
>
> <p dir="rtl">One <b>two</b> <i>three</i> four.</p>
>
> It has some interesting selection behavior.
>
OK, in fact I tried something like: "abc<b dir=rtl>CDE</b>fgh", because the
dir attribute can be applied to almost all tags. I tried your example, and
it seems that the the is some "bizare" mechanism which is similar to what we
want to bidi. This is a good thing.
> > > I agree that it would be cleaner if we could do the reordering and
> > > shaping in all cases, but if it turns out to be very difficult to
> > > circumvent the platform's reordering and/or shaping on some platforms,
> > > why not let those platforms do it?
> >
> > One reason is that the reordering algorithm of Windows 95/98 is
different
> > than the Unicode one.
>
> Ouch. Well, if it's not too difficult to bypass the platform's
> reordering, then let's do so, but if it *is* very difficult, maybe our
> initial version should just use the Windows algorithm.
The problem has two facets :
1) Windows 95/98 Reordering <> Unicode Reoredering
2) Windows NT/2000 Reordering == Unicode Reordering
And we need to implement Unicode algorithm, because this is the standard.
> Does IE implement
> the Unicode bidi spec correctly, even on the various Windows versions?
IE itself uses Unicode reordering even on Win95/98.
>
> > We can also try to see how complicated is to do the measurement ourself
by
> > using the metrics. Except the kerning (which is in a simple table of the
> > font), what can restrain us to do simples additions ?
>
> Kerning is one thing that we would need to keep in mind. Another thing
> is speed. Text measurement is critical. Also, we would need to think
> about the units we use internally. If we compute measurements in pixels,
> the numbers may be rounded too much. The TrueType font may have better
> (larger) numbers to work with, in terms of "units per em" (often 2048).
OK, but I think that from a speed point of view, the best is to do ourself
the calculations based on the info from
the TrueType font.
So, there are two solutions now:
I]
a) DrawString: ETO by using the glyphs.
b) GetWidth: ETO in a in-memory DC.
II]
a) DrawString: ETO by using the glyphs. (as in I]).
b) GetWidth: Doing ourself the additions using the "large" units from the
font, and using the kernel tables.
>
> Erik
>
Yaacov Akiba
sl...@il.ibm.com
Hello!
I have no experience with Mac, but
1) For shaping in Arabic, if one use the FE Unicode range no shaping occurs
since the text is already
shaped (in contrast with the 06 range).
2) For reordering, if Mac uses Unicode reordering, it has to understand LRO or
at least LRM. If not,
we can always cut the string in runs of strong RTL chars reversed, and non
strong RTL chars.
Yaacov Akiba
ft...@netscape.com (Yung-Fong Tang) on 14/12/99 00:38:59
Please respond to ft...@netscape.com (Yung-Fong Tang)
To: Yaakob Slama/Israel/Contr/IBM@IBMIL
cc: ped...@apple.com
Subject: Re: layout engine, text, content model, etc
Yaacov Akiba Slama wrote:
Great..
Agree
Agree.
> Erik van der Poel <er...@netscape.com> wrote in message
Erik van der Poel
>
> OK, but I think that from a speed point of view, the best is to do ourself
> the calculations based on the info from the TrueType font.
> So, there are two solutions now:
> I]
> a) DrawString: ETO by using the glyphs.
> b) GetWidth: ETO in a in-memory DC.
One possible variant of this is:
I.2]
a) GetWidth: ETO in off-screen DC
b) DrawString: copy off-screen DC to screen
But it might still be too slow, since GetWidth is called more often than
DrawString.
> II]
> a) DrawString: ETO by using the glyphs. (as in I]).
> b) GetWidth: Doing ourself the additions using the "large" units from
> the font, and using the kernel tables.
I don't know if it's actually "correct" to use the large units from the
font itself. I think we should compare our results with GTE on
appropriate fonts and platforms. I guess the point I was trying to make
is that we need to round the computations carefully.
Also, just out of curiosity, how does one do kerning quickly, in a
reasonable amount of memory?
Erik
Frank Tang
sl...@il.ibm.com wrote:
> Hello!
>
> I have no experience with Mac, but
> 1) For shaping in Arabic, if one use the FE Unicode range no shaping occurs
> since the text is already
> shaped (in contrast with the 06 range).
> 2) For reordering, if Mac uses Unicode reordering, it has to understand LRO or
> at least LRM. If not,
> we can always cut the string in runs of strong RTL chars reversed, and non
> strong RTL chars.
You are asumming that Mac use Unicode. Howerver, Mac QuickDraw does not handle
Unicode. ATSUI handle Unicode. We currently use TEC (Text Encoding Converter) to
convert Unicode to Arabic Script and use QuickDraw to render it. The TEC does
understand LRO or LRM, but I am not sure it understand Presentation Form B.
I still do not understand why we want to do shaping and reordering code in
cross-platform code instead of in platform code. Why ?
Deborah Goldsmith
<ft...@netscape.com> wrote:
> I don't think you have a way to do that on the Mac. I think it is easy to do
> that on X, but I don't think you can find a way to by-pass reordering/shaping
> code on Mac.
After a quick check, we don't think it's possible to do that in
QuickDraw Text in a supported way. It can be done in ATSUI by using the
Unicode bidirectional override characters and turning off linguistic
shaping as a font feature.
I hope this helps...
--
Deborah Goldsmith
Manager, International Toolbox Group
Apple Computer, Inc.
gold...@apple.com
sl...@il.ibm.com
Hello !
First, one fundamental reason for implementing reordering in cross platform code
is that it needs to be implemeted
in the layout phase.
I am describing here a very simplified version of the reordering algorithm to
explain why.
1) There are 3 types of chars: LTR, RTL and N (neutrals).
2) For each char in a string there is a "Environment" direction (which is the
dir of the HTML of one specified by dir=??).
3) We need to determine the dir of each N. There are two cases XN...NX or
XN...NY where X,Y equals LTR or RTL and
X<>Y. In the first case the dir of N is X, in the second it is the dir of the
"Environement".
4) This algorithm is applied to an entire paragraph before breaking it and even
if there are changes of style, fonts etc.. in it.
5) The last thing to do is the actual reodering wich reorder each run of RTL
chars (some could be born N).
This can be done at a latter stage if we carry the dir info until then (in fact
instead of dir, we have a level).
The 4) implies that we need to do reordering INTO the layout.
Second, the reasons for implementing shaping in cross platform code are (at
least):
- The code is written once and will work for every platform, assuming we are
disabling the reordering/shaping
of the platform. I understand that this point is not relevant for a platform
which support bidi, but since there are
far more platform without bidi than with bidi.
- There is no problem of synchronisation between code.
Third, if the mac cannot render FE-range chars, if it understands joiners (which
is a condition for a good bidi support), we can replace each FE-range char by a
serie Joiner,06-range char,Joiner.
Yaacov.
ft...@netscape.com (Yung-Fong Tang) on 15/12/99 02:17:10
Please respond to ft...@netscape.com (Yung-Fong Tang)
To: Yaakob Slama/Israel/Contr/IBM@IBMIL, ped...@apple.com
cc:
sl...@il.ibm.com wrote:
> I don't think you have a way to do that on the Mac. I think it is easy to do
> that on X, but I don't think you can find a way to by-pass reordering/shaping
> code on Mac.
>
Roozbeh Pournader
On Wed, 15 Dec 1999, sl...@il.ibm.com wrote:
> I am describing here a very simplified version of the reordering algorithm to
> explain why.
> 1) There are 3 types of chars: LTR, RTL and N (neutrals).
I know about simplified BIDIs, like ones implemented in Iranian standards
before Unicode. I really wonder what should we do with ARABIC-INDIC
digits, as called by Unicode. In simplied BIDIs, they are of another type,
and their behaviour is different from normal LTR. E.g., if you are
including an Arabic quotation in English text, an Arabic-indic digit will
not mark the end of quotation although LTR, while a Latin letter will.
> - The code is written once and will work for every platform, assuming we are
> disabling the reordering/shaping
> of the platform. I understand that this point is not relevant for a platform
> which support bidi, but since there are
> far more platform without bidi than with bidi.
There are also bugs with shaping. As an example, Windows 9x has shaping
bugs with ARABIC LETTER FARSI YEH, not solved yet.
--roozbeh
Frank Tang
sl...@il.ibm.com wrote:
Hello !First, one fundamental reason for implementing reordering in cross platform code
is that it needs to be implemeted
in the layout phase.I am describing here a very simplified version of the reordering algorithm to
explain why.
1) There are 3 types of chars: LTR, RTL and N (neutrals).
2) For each char in a string there is a "Environment" direction (which is the
dir of the HTML of one specified by dir=??).
3) We need to determine the dir of each N. There are two cases XN...NX or
XN...NY where X,Y equals LTR or RTL and
X<>Y. In the first case the dir of N is X, in the second it is the dir of the
"Environement".
4) This algorithm is applied to an entire paragraph before breaking it and even
if there are changes of style, fonts etc.. in it.
5) The last thing to do is the actual reodering wich reorder each run of RTL
chars (some could be born N).
This can be done at a latter stage if we carry the dir info until then (in fact
instead of dir, we have a level).The 4) implies that we need to do reordering INTO the layout.
I agree w/ you that you need to resolve (decide) the directionality of
each character in the layout stage.
Howerver, it does not mean you need to reorder or shape the characters
in the layout stage, right ?
If you can resolved LTR N and RTL text into resolvedLTR and resolvedRTL
text run, then you can do shaping and reordering in the platform specific
code.
What you can do is the following- Assuming lower case is LTR and UPPER
case is RTL and '.' is weak.
THIS IS A TEST. i love arabic.
and the global direction is left to right-
The the cross-platform code could resolve the directionality into two
text run
RTL [THIS IS A TEST.] LTR [i love arabic.]
and then you can lay them down word by word
RTL [THIS] RTL [IS] RTL [A] RTL [.TEST]
LTR [i] LTR [love] LTR [arabic.]and the paltform specific code can resolve them into
[SIHT] [SI] [A] [.TEST]
[i] [love] [arabic.]
.And it show to the screen in the following seq
SIHT SI SIHT A SI SIHT .TEST A SI SIHT .TEST A SI SIHT i .TEST A SI SIHT i love .TEST A SI SIHT i love arabic.If the global direction is rigth to left , then it show the screen in the following seq
SIHT SI SIHT A SI SIHT .TEST A SI SIHT i .TEST A SI SIHT i love .TEST A SI SIHT i love arabic. .TEST A SI SIHT
Second, the reasons for implementing shaping in cross platform code are (at
least):- The code is written once and will work for every platform, assuming we are
disabling the reordering/shaping
I agree w/ you the reordering/shapping code should be implemented in corss-platform code. But I am not sure it is necessary to be CALLED from cross platform code.
of the platform. I understand that this point is not relevant for a platform
which support bidi, but since there are
far more platform without bidi than with bidi.
- There is no problem of synchronisation between code.Third, if the mac cannot render FE-range chars, if it understands joiners (which
is a condition for a good bidi support), we can replace each FE-range char by a
serie Joiner,06-range char,Joiner.
It is very easy to find out today. just put þxx; and use Mozilla to view by using the current mozilla.
Erik van der Poel
>
> I agree w/ you that you need to resolve (decide) the directionality of
> each character in the layout stage.
> Howerver, it does not mean you need to reorder or shape the characters
> in the layout stage, right ?
> If you can resolved LTR N and RTL text into resolvedLTR and
> resolvedRTL text run, then you can do shaping and reordering in the
> platform specific code.
I believe we have been contrasting the layout stage with the drawing
stage, and the cross-platform (XP) code with the platform-specific (PS)
code. But the layout stage involves both XP and PS code, because you
need to measure text to lay it out, and you need to call platform APIs
to measure text. Similarly, the drawing stage also involves both XP and
PS code.
Now let's analyze the problem to see if we need to do any reordering or
shaping in the layout stage.
For Mozilla, we are trying to implement several specs on several
platforms. In this case, the spec is the Unicode bidi algorithm. Let's
choose Windows as the platform for now. (We'll look at the other
platforms later, but choose to look at Windows only for now, to keep the
discussion simple.)
Yaacov claims that the Win9x character-based APIs do not reorder glyphs
in the same way as the Unicode bidi algorithm. Let's assume that he is
right. We can get around this problem by using ExtTextOut with the
ETO_GLYPH_INDEX flag, passing the text as glyph codes. However, there is
no corresponding API to measure text passed as glyph codes. Let's assume
that we will write our own routine to measure text, based on font
metrics that we retrieve from the font.
In order to measure text, we must map Unicode character codes to font
glyph codes. To map Arabic character codes to glyph codes, we must
perform shaping. So, shaping must be performed in the layout stage.
(Does Arabic also have compulsory ligatures? If so, we will sometimes
map 2 or more Unicodes to a single glyph code.)
Now let's look at the measurement itself. To measure the width of a
number of glyphs laid out along a line, we must add all of their
"strides". A stride is the increment in the X coordinate for a
particular glyph. As we add up all of the strides, we must also take
kerning into account, which can be either positive, zero or negative for
each pair of glyphs. I assume that kerning pairs are in visual order,
but we can always reverse the way we look at this, depending on the
direction of the current run (string). So, I believe we do not need to
reorder glyphs to measure them. (Actually, we must *not* reorder any
parts of a paragraph before breaking it into lines.)
(By the way, if there is a space between LTR and RTL text, where is the
space "drawn"? Near the end of the first logical part, or near the
beginning of the next logical part? I suppose the Unicode spec has rules
about this.)
Now let's look at the drawing stage. It is pretty obvious that both
shaping and reordering must be performed.
Next, let's see whether shaping and reordering must be performed in XP
or PS code. Shaping involves mapping character codes to glyph codes, and
glyph codes are platform-specific, so shaping must be performed in PS
code. (The character to glyph mapping can be performed in 2 steps: (1)
map Arabic Unicode character codes to Unicode presentation forms, and
(2) map Unicode presentation forms to font glyph codes. But if we are
going to cache the "output" buffer, I'd rather cache the glyph codes,
not the presentation forms, so that we actually use memory to save some
computation, rather than using memory *and* still performing extra
computation.)
Reordering involves reversing the order of an array. We can either
reverse an array of Unicodes, or an array of the platform API's codes.
If we reverse Unicodes, we can do this in XP code. But we must be
careful to do this in such a way that we break lines at the right
points.
Now let's come back to the Mac. If it is true that the Mac always
reorders RTL text and there is no way around that API, then it would be
wasteful if Mozilla reordered the text once in XP code and again in
Mac-specific code before calling the Mac API. This would seem to argue
that we should reorder in PS code, so that we don't end up with wasteful
triple reordering on the Mac. If we reorder in PS code, we reorder once
on all platforms. (ATSUI may have some way to avoid reordering, possibly
by passing glyph codes, but we have found problems in ATSUI. And
QuickDraw GX is not available on all Macs.) However, the Mac reordering
may not comply with the Unicode spec (e.g. LRO, RLO?). In this case, we
may have to split the text into smaller chunks, and perform some of the
reordering ourselves. Or use the LRM/RLM trick that Yaacov mentioned
earlier, if that works on the Mac.
Since we have to map Unicodes to platform API codes, and since we have
to reorder text in some cases, it seems like we could boost performance
by caching the results of these computations. One way to do this would
be to compute the results while measuring text, and to reuse the results
when drawing the text.
Currently, the layout engine measures text one word at a time, but draws
text one line at a time. I don't know if this is accurate, actually.
What happens if there is a kerning pair involving the space character?
Or does that never occur?
It might be better if we pass more than a word to the measuring routine.
We could pass a bunch of words, together with the points at which the
line may be broken, and the width of the current area. Then the routine
would simply stop when it reached the break point just past the width,
and return info about where it stopped. It could also return reordered
glyph codes, to be used by the drawing routine later.
But perhaps we would want to do this project in a number of smaller
steps. For example, maybe we would not try to cache the glyphs in the
first stage. Also, we may want to keep the current layout algorithm of
measuring one word at a time for now.
How does this sound?
Erik
Erik van der Poel
It's nice to "see you" in this newsgroup/mailing list. Thanks for taking
the time to investigate and respond.
Deborah Goldsmith wrote:
>
> In article <38557582...@netscape.com>, Frank Yung-Fong Tang
> <ft...@netscape.com> wrote:
>
> > I don't think you have a way to do that on the Mac. I think it is easy to do
> > that on X, but I don't think you can find a way to by-pass reordering/shaping
> > code on Mac.
>
> After a quick check, we don't think it's possible to do that in
> QuickDraw Text in a supported way. It can be done in ATSUI by using the
> Unicode bidirectional override characters and turning off linguistic
> shaping as a font feature.
I also asked about reordering, but I guess shaping must also be
considered, since it involves looking at neighboring characters to
decide which shape to choose. The initial and final glyphs are of course
highly dependent on finding the correct beginning and end of the word,
which would be different if we performed reordering ourselves.
So, it sounds like we must pass unreordered and unshaped text to
QuickDraw.
Deborah, I heard from Frank that he has let someone at Apple know about
our initial experiments with ATSUI. Do you know if there are plans to
address the problems we had? Perhaps they've already been solved in more
recent versions of ATSUI? I'm referring to the speed and ugly scaling
problems.
We'd love to use ATSUI, of course, since it takes Unicode. Maybe we used
it in the wrong way, and should have called different APIs, or called
them differently? Might be worth it to investigate further. Frank?
Erik
Deborah Goldsmith
> Deborah, I heard from Frank that he has let someone at Apple know about
> our initial experiments with ATSUI. Do you know if there are plans to
> address the problems we had? Perhaps they've already been solved in more
> recent versions of ATSUI? I'm referring to the speed and ugly scaling
> problems.
>
> We'd love to use ATSUI, of course, since it takes Unicode. Maybe we used
> it in the wrong way, and should have called different APIs, or called
> them differently? Might be worth it to investigate further. Frank?
The speed problems are partly problems in ATSUI (which we are progressively
addressing in successive Mac OS releases), but also partly a consequence of
creating and destroying text layout objects every time a string is drawn. It
is much more efficient to create the necessary text layout and style objects
once, and replace the text when it is time to draw something new.
ATSUI definitely needs more performance work, but I have ATSUI-based
programs that have perfectly fine performance for documents the size of
typical web pages and larger. We would have to work more closely with you to
determine if it is possible to speed up the ATSUI version of Mozilla.
Things that are slower than we like right now:
1. Opening large (>50K) documents. This can be ameliorated by doing layout
during idle time. Unlike QuickDraw, ATSUI supports full Unicode layout,
which takes more CPU time. We plan to focus our performance tuning efforts
heavily on this.
2. Using large repertoire (e.g., Unicode) fonts. Some caches are too small
to hold the tables in these fonts, and thrash. We plan to work on this.
3. Iterating through installed fonts. Again, this is a cache-thrashing issue
we know how to address.
We plan to devote considerable effort to addressing these performance
problems in the near future. We did not get a good sense for what areas of
ATSUI needed performance work until we had a major client, which was the new
Multilingual Text Editor (MLTE) in Mac OS 9. We have already made major
performance improvements in ATSUI in Mac OS 9 in the course of getting MLTE
out the door. You might try implementing a simple application using MLTE
(it's easier to use than Text Edit) to get a sense of what ATSUI's
performance can be like when used properly.
The problems with display of small sizes of some fonts is due to the fact
that ATSUI doesn't support NFNT bitmaps, because their repertoire is limited
to MacRoman and other one byte character sets. Fonts that have bitmaps in
the sfnt work fine with ATSUI. ATSUI and QuickDraw both use exactly the same
rendering path. If you removed the NFNT bitmaps from Times etc., and turned
on fractional spacing, you would see the same results in QuickDraw as in
ATSUI. It is possible to turn off fractional spacing in ATSUI if that is
desired.
At this point we are examining possible solutions to this problem. We are
leaning towards putting a new, metric-compatible set of bitmaps into the
'sfnt' portion of our standard fonts. ATSUI would use these, and QuickDraw
would continue to use 'NFNT' bitmaps for compatibility of old documents.
However, we haven't decided for sure. There is a bug written against this,
and I consider it a high priority bug to address.
Part of the problem is that the current bitmaps in our standard Roman fonts
are very disproportionate. The 9 point fonts in particular are much larger
than a 9 point scaled version of the outline. There is really not much
reason for this except as a historical oddity we must preserve for
compatibility in QuickDraw. If we do new bitmaps for ATSUI, they are likely
to be smaller than the NFNTs, because we want the metrics to be compatible
with the outlines (having the metrics be so far off causes real WYSIWYG
problems in printing). One workaround in ATSUI is just to use larger point
sizes; you get the effect you want on the screen, and the metrics are much
closer to what you will get when you print.
I would be happy to continue to work with you and others to try to
understand your needs and to offer advice on how best to use ATSUI, and all
of our other international technologies.
Sincerely,
Erik van der Poel
>
>
> Separation of the input text into paragraphs. The rest of the
> algorithm affects only the text between paragraph separators.
>
> Resolution of the embedding levels of the text. In this phase,
> the directional character types, plus the explicit format
> codes, are used to produce resolved embedding levels.
>
> Reordering the text for display on a line-by-line basis using
> the resolved embedding levels, once the text has been broken
> into lines.
>
> So, I think the 3rd phase should be performed *after* the text is split
> into lines.
>
> The 1st phase clearly occurs when we parse the document and create the
> content model.
Actually, the bidi algorithm must be applied to block elements (as
opposed to inline elements). However, an element's block-ness is not
determined by the element itself, but by the style sheet applied to it.
CSS has a property called "display", which can be block, inline or some
other value. I tested this in Mozilla just now. If you set a <B>
element's display to block, Mozilla displays it as a separate block
element, rather like a paragraph.
So, I think the bidi algorithm's 1st and 2nd phases should be performed
while building the frame model from the content model and the style
sheets, and the 3rd phase should be performed while displaying the text.
Erik
Erik van der Poel
>
> 1) For shaping in Arabic, if one use the FE Unicode range no shaping
> occurs since the text is already shaped (in contrast with the 06 range).
According to a Unicode expert (see attached message), Unicode's FE range
isn't very good for Arabic. Would it be better to use the GSUB table in
TrueType Open and OpenType? Or is the GSUB table missing in some Arabic
fonts?
Erik
Roozbeh Pournader
On Sun, 9 Jan 2000, Erik van der Poel wrote:
> According to a Unicode expert (see attached message), Unicode's FE range
> isn't very good for Arabic. Would it be better to use the GSUB table in
> TrueType Open and OpenType? Or is the GSUB table missing in some Arabic
> fonts?
They are also not enough. I checked the FE range in depth recently, and I
found many glyph shapes missing.
I know that Arabic TrueType Open and OpenType fonts encode the glyph
shapes using their "FE" code, and using GSUB to point to that, but I think
there were Arabic TrueType fonts before introduction of TrueType
Open. BTW, in case of X, I think most of Arabic fonts use "FE" codes, so
we need that.
--Roozbeh
Roozbeh Pournader
In my search for glyph shapes in FB50--FBCF and FE70--FEFF Unicode ranges,
I found the glyph shapes for following characters missing:
0672--0676
0678
067C
067D
0681
0682
0685
0689--068B
068F
0692--0697
0699--06A3
06A5
06A7
06A8
06AA--06AC
06B0
06B2
06B4--06B7
06BC
06BD
06C2--06C4
06CA
06CD
06CE
06D1
So they are more than ignorable. But again they are the less frequenly
used ones, which to my knowledge are rarely used. From the info in the
comments for each character, the existing characters are enough for
rendering Arabic, Persian and Urdu. But not if you want to do Pashtu,
Kurdish, Uighur, Sindhi, etc.
--Roozbeh
Erik van der Poel
>
> I know that Arabic TrueType Open and OpenType fonts encode the glyph
> shapes using their "FE" code, and using GSUB to point to that
I haven't actually looked at any GSUB tables in Arabic fonts, but the
GSUB spec says that the GSUB table maps *glyphs* to other glyphs, i.e.
not characters to glyphs:
http://www.microsoft.com/typography/OTSPEC/gsub.htm
My understanding of the spec is that Arabic fonts should have a cmap
that maps each character in the 06 range to *one* of the possible
glyphs, and that the GSUB table then maps that glyph to one of several
other glyphs, depending on context (beginning, middle or end of word).
Of course, the GSUB mapping is not always one-to-one, for example when
ligatures are involved.
> but I think
> there were Arabic TrueType fonts before introduction of TrueType Open.
Should Mozilla support those older TrueType fonts? How common are they
today? Do they use Unicode cmaps?
> BTW, in case of X, I think most of Arabic fonts use "FE" codes, so
> we need that.
Aren't most of the Arabic fonts on X single-byte? I thought they used
encodings such as iso8859-6. If so, they don't actually use codes like
0xFEXX, right? Or did you mean that they use presentation forms
(glyphs), rather than characters?
Erik
Franck Portaneri
>
>
> > BTW, in case of X, I think most of Arabic fonts use "FE" codes, so
> > we need that.
>
> Aren't most of the Arabic fonts on X single-byte? I thought they used
> encodings such as iso8859-6. If so, they don't actually use codes like
> 0xFEXX, right? Or did you mean that they use presentation forms
> (glyphs), rather than characters?
>
- Single bytes : http://www.langbox.com/AraMosaic/mozilla/fontXFE
- Unicode encoded : http://crl.nmsu.edu/~mleisher/arabic24.html
The single byte fonts cannot just follow the iso8859-6 character layout - this has
no sense, no way to display arabic text contexted with only the "isolated glyphs"
characters shown in iso8859-6. So additional 0xFEXX glyphs have been added
to free position in single byte fonts.
But as far as I know, there is no standard to encoding additional glyphs in
these fonts and each implementation uses it own mapping.
We propose the iso8859-6.8X (http://www.langbox.com/arabic/fontara8X.html )
freely used with AraMosaic and with the Axmedit for Linux.
On older systems we even were obliged to use only 96 glyphs (iso8859-6.8 at
http://www.langbox.com/arabic/fontara8.html ) due to device limitation reason,
but X can display all 8 bit code.
I think that Sun also adopted single byte fonts that follow iso8859-6.8
(or nearly) too - (Sun Ireland, any input...)
Single bytes fonts are easier (and faster) to use, but not all 0xFEXX can
be encoded and so compromises are done (i.e. one glyph is used for several
0xFEXX glyphs) - For X Display this is acceptable, but for printing it
is preferable to have all 0xFEXX glyphs available.
For the Unicode layout, Mark Leisher did an big work too and provides 630+
Arabic glyphs in his 24 points fonts - He even provides the xmbdfed editor.
Franck
--
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Franck Portaneri mailto:fra...@langbox.com
LangBox International
Immeuble Space, Batiment B Tel: +33 (0)4 9371 1410
208-212, Route de Grenoble Fax: +33 (0)4 9371 1560
06200 Nice - France http://www.langbox.com
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Roozbeh Pournader
On Mon, 10 Jan 2000, Erik van der Poel wrote:
> I haven't actually looked at any GSUB tables in Arabic fonts, but the
> GSUB spec says that the GSUB table maps *glyphs* to other glyphs, i.e.
> not characters to glyphs:
>
> http://www.microsoft.com/typography/OTSPEC/gsub.htm
>
> My understanding of the spec is that Arabic fonts should have a cmap
> that maps each character in the 06 range to *one* of the possible
> glyphs, and that the GSUB table then maps that glyph to one of several
> other glyphs, depending on context (beginning, middle or end of word).
I'm sorry I've not read the spec yet, but only took a look. I will comment
when I read that carefully.
> Should Mozilla support those older TrueType fonts? How common are they
> today? Do they use Unicode cmaps?
I can only answer that about Persian fonts, also after asking the local
experts (!).
>
> > BTW, in case of X, I think most of Arabic fonts use "FE" codes, so
> > we need that.
>
> Aren't most of the Arabic fonts on X single-byte? I thought they used
> encodings such as iso8859-6. If so, they don't actually use codes like
> 0xFEXX, right? Or did you mean that they use presentation forms
> (glyphs), rather than characters?
I meant Unicode-enabled fonts. I don't know much enough about Arabic 8-bit
X fonts. Persian X fonts don't use the single character/letter rule of
Unicode in anyway. You should know the font glyph encoding model before,
which is usually IRAN SYSTEM
(http://www.sharif.ac.ir/~roozbeh/farsiweb/iransystem.txt).
--Roozbeh
Roozbeh Pournader
On Mon, 10 Jan 2000, Franck Portaneri wrote:
> We propose the iso8859-6.8X (http://www.langbox.com/arabic/fontara8X.html )
> freely used with AraMosaic and with the Axmedit for Linux.
> On older systems we even were obliged to use only 96 glyphs (iso8859-6.8 at
> http://www.langbox.com/arabic/fontara8.html ) due to device limitation
> reason, but X can display all 8 bit code.
> I think that Sun also adopted single byte fonts that follow iso8859-6.8
> (or nearly) too - (Sun Ireland, any input...)
Does Langbox use any Persian glyph encodings for their PMosaic? As I know,
there are only two X font encodings used in Persian, one the same as IRAN
SYSTEM (I will post the URL of one of the free ones later) which is has
two glyphs for most of the letters, and four for a few (against Unicode
which uses four for most of them and two for a few), so not suitable for
typography, but only for computer display. The other is called FSFE which
is four glyphs per letter, and I can post a description in there's a need.
Both have been invented by companies, and not standard organizations.
FSFE is rarely used but it fully supports both Persian and Arabic except
for ARABIC KAF (vs. PERSIAN KAF which is named ARABIC LETTER KEHEH in
Unicode). IRAN SYSTEM lacks many many more.
--Roozbeh
Franck Portaneri
>
> On Mon, 10 Jan 2000, Franck Portaneri wrote:
>
> > We propose the iso8859-6.8X (http://www.langbox.com/arabic/fontara8X.html )
> > freely used with AraMosaic and with the Axmedit for Linux.
> > On older systems we even were obliged to use only 96 glyphs (iso8859-6.8 at
> > http://www.langbox.com/arabic/fontara8.html ) due to device limitation
> > reason, but X can display all 8 bit code.
> > I think that Sun also adopted single byte fonts that follow iso8859-6.8
> > (or nearly) too - (Sun Ireland, any input...)
>
> Does Langbox use any Persian glyph encodings for their PMosaic?
You meant AraMosaic? AraMosaic supports ISIRI 3342 encoding, but doesn't include
the 6 additional letters glyphes in the font - So it just convert them to their
nearest glyphs in the iso8859-6.8X layout - This is not correct, and this
is just a "patched" way to read Persian text (as they seems to do with CP1256
windows).
I would be interested by your two X font encodings used in Persian...