SGML question
Anders Thulin
I've been trying to make sense of the SGML standard. I'm beginning to
think that it shouldn't be done at home, and only attempted by highly
trained professionals :-,
The first problem is with the modes used for recognizing delimiters
(section 9.6.1 in the standard). Here is the background:
1) SGML delimiters are recognized only in certain modes. For example,
"<?" is recognized as the PIO delimiter only in the CON and DSM
modes. In other modes "<?" is not regarded as a delimiter.
2) According to section 9.6.1, CON mode delimiters are recognized in
the _content_ rule of the SGML grammar, and in the _marked section_
of marked section declarations that occur in _content_.
This seems to be reasonably clear and unambiguous.
3) Annex F (not formally a part of the standard) says that the
document is scanned in CON mode after the prolog.
This fits reasonably well with the previous rule.
This is my problem. Section 9.6.1 seem to be very clear on the extent
on CON mode: it's essentially used `inside' the _content_ production
[27]. How, then, are delimiters *outside* _content_ recognized?
After folding some irrelevant(?) productions, I find the syntax for a
SGML document entity begins like this:
s*, SGML declaration, prolog, s*, start-tag? content, end-tag?
^CON mode starts here
^or here (Annex F)
These two 'points' are by and large the same: both s* and start-tag
may be empty. No problem so far.
Now, _prolog_ may contain processing instructions, which begin with
the PIO delimiter ("<?"). PIO is recognized only in CON and DSM modes.
And DSM cannot be used in this context - it is used only in
declaration subsets or in marked sections.
Hence, the document must be scanned in CON mode at the *beginning* of
_prolog_. Using the same argument, CON mode recognition must be used
also at the beginning of the document for the SGML declaration.
This seem to be a contradiction with rule 2 above: CON mode
recognition *must* be used also outside of _content_, even though 9.6.1
states the CON delimiters are recognized in _content_ and in the
_marked section_ of ... and so forth.
Is there a contradiction here, or have I missed something important?
---
Another problem is the ">" delimiter. According to the table in Figure
3 (page 31) in the standard, ">" is recognized as either MDC or TAGC
in CTX mode. But I find nothing that says why it should be recognized
as one rather than the other.
I would assume that ">" would be recognized as TAGC if the parser was
currently parsing a _tag_, and as MDC in markup declarations. And this
is the behaviour I would expect from any decent implementation. I just
can't see that it is supported by the standard.
So: would a scanner that recognized ">" as MDC when the person who
created the document meant TAGC strictly speaking be in error?
--
Anders Thulin a...@prosys.se {uunet,mcsun}!sunic!prosys!ath
Telesoft Europe AB, Teknikringen 2B, S-583 30 Linkoping, Sweden
Larry Jones
> I've been trying to make sense of the SGML standard. I'm beginning to
> think that it shouldn't be done at home, and only attempted by highly
> trained professionals :-,
<flame>
It shouldn't be done at all. The SGML standard is, without a doubt,
the poorest excuse for a language standard I have ever seen. I can
only assume that it was developed by people with no knowledge of
formal languages who adopted a formalism to make the resulting
document look more technical. Regular expressions are clearly
inappropriate for describing the language, a phrase structure grammar
would have been much better. In addition, the standard contains a
number of errors, inconsistencies, ambiguities, and violations of
the formalisms. The SGML standard is in dire need of interpretation
and revision -- unfortunately, unlike ANSI standards which clearly
specify how to go about submitting comments and requesting
interpretations, ISO standards provide no clues at all.
</flame>
Now, having gotten that out of my system, allow me to try to provide
some answers to your questions.
> This is my problem. Section 9.6.1 seem to be very clear on the extent
> on CON mode: it's essentially used `inside' the _content_ production
> [27]. How, then, are delimiters *outside* _content_ recognized?
That's what it says, but it's not what it means. CON mode is simply
the default mode when you're not in any other mode. So you do indeed
start off at the very beginning of the entire document in CON mode
and return to it whenever a nested mode is ended.
> Another problem is the ">" delimiter. According to the table in Figure
> 3 (page 31) in the standard, ">" is recognized as either MDC or TAGC
> in CTX mode. But I find nothing that says why it should be recognized
> as one rather than the other.
CXT isn't a real mode, it's just a pseudo-mode. Some delimiters
aren't recognized as such unless they are followed by some particular
context as sepcified in 9.6.2. This context may include other
delimiters so CXT mode is used to indicate that a delimiter needs to
be recognize while verifying the context of another delimiter. Thus,
there should really be a separate CXT mode for each of the contexts
listed in 9.6.2 (e.g. DCL-CXT, GI-CXT, etc.) and the table in Figure
3 should be expanded appropriately (e.g. TAGC should be GI-CXT [but
only if "SHORTTAG YES" is specified on the SGML declaration], and
MDC should be DCL-CXT and MSE-CXT).
----
Larry Jones UUCP: uunet!sdrc!thor!scjones
SDRC scj...@thor.UUCP
2000 Eastman Dr. BIX: ltl
Milford, OH 45150-2789 AT&T: (513) 576-2070
Oh, now YOU'RE going to start in on me TOO, huh? -- Calvin
Anders Thulin
>In article <5...@helios.prosys.se>, a...@prosys.se (Anders Thulin) writes:
>> I've been trying to make sense of the SGML standard. I'm beginning to
>> think that it shouldn't be done at home, and only attempted by highly
>> trained professionals :-,
>
><flame>
>number of errors, inconsistencies, ambiguities, and violations of
>the formalisms.
Glad do hear it ... sort of ... well, ... :-/
Is there any list of these errors, etc floating around? It would
probably be a help to the readers of comp.text.sgml to have one
around. Alternatively, can we compile one? I'm thinking of sending
the CON and CTX questions to ISO for formal interpretation. A list
of bugs in the standard might provide sufficient reason to revise
it.
Or perhaps this has been tried earlier without success?
Anders Thulin
>> I've been trying to make sense of the SGML standard. I'm beginning to
>> think that it shouldn't be done at home, and only attempted by highly
>> trained professionals :-,
>
><flame>
>number of errors, inconsistencies, ambiguities, and violations of
>the formalisms.
Glad to hear it ... sort of ... well, ... :-/
Is there any list of these errors, etc floating around? It would
probably be a help to the readers of comp.text.sgml to have one
around. Alternatively, can we compile one? I'm thinking of sending
the CON and CTX questions to ISO for formal interpretation. A list
of bugs in the standard might provide sufficient reason to revise
it.
Or perhaps this has been tried earlier without success?
--
Larry Jones
> Is there any list of these errors, etc floating around? It would
> probably be a help to the readers of comp.text.sgml to have one
> around. Alternatively, can we compile one? I'm thinking of sending
> the CON and CTX questions to ISO for formal interpretation. A list
> of bugs in the standard might provide sufficient reason to revise
> it.
I haven't seen any official list, although I have seen a very nice
paper written by someone at Ohio State University (I think) that
pointed out most of the problems I was aware of. I'll see if I
can dig out a copy and post some or all of it. If not, I'd be
glad to try to remember the problems, but it's been a couple of
years since I worked on an SGML parser so my memory is apt to be
a bit faulty.
----
Larry Jones UUCP: uunet!sdrc!thor!scjones
SDRC scj...@thor.UUCP
2000 Eastman Dr. BIX: ltl
Milford, OH 45150-2789 AT&T: (513) 576-2070
Temporary insanity! That's all it was! -- Calvin
David J Birnbaum
> Is there any list of these errors, etc floating around? It would
^^^^^^^^^^^^^^
Does comp.text.sgml exist? I read the above posting in comp.text and
my system doesn't know about any comp.text.sgml group. Should I bug
my system manager about it? Is it a group that is being planned but
hasn't started yet? Is this a typo for comp.text?
Thanks,
David
=======================================================================
David J. Birnbaum
d...@wjh12.harvard.edu [Internet]
d...@harvunxw.bitnet [Bitnet]
Edward Vielmetti
Does comp.text.sgml exist? I read the above posting in comp.text and
my system doesn't know about any comp.text.sgml group. Should I bug
my system manager about it? Is it a group that is being planned but
hasn't started yet?
It is a group which has passed the usenet voting procedure but hasn't
yet been created -- there's a prescribed waiting period of a few days
to let anyone object etc. The annoucement really should hit
"news.announce.newgroups" first before the group is created.
--Ed
Anders Thulin
>3. It doesn't deal with graphics, tables, or equations (is that 5 strikes?).
Graphics, no. Although graphics should probably be treated as non-SGML
data, which SGML can handle.
Tables it does provide for. That is, SGML permits you to mark up a
table. The actual formatting is left to a 'processor', which is
supposed to do what you want in some way. So, provided that you have
this 'processor' you can do tables. I understand the word 'processor'
to mean a formatting program, e.g. troff.
Equations is almost the same thing. SGML permits text (entities?) to
be written in a 'notation', which then is interpreted by some
'notation translator'. I believe there is a registered BSI notation
standard which could be used. Alternatively, TeX notation could
probably be used as well.
You are right that SGML doesn't provide these capabilities directly.
But a SGML parser/translator should not choke on them either. It
should just pass the buck to something who does better.
> It only provides text portability, which ASCII does more elegantly.
ASCII isn't much of a help if I want to insert a Swedish { in the
text. (That '{', of course, is an 'a' with an umlaut accent). Nor is
ASCII very good at indicating italics or boldface.
Some further coding conventions are required if I want to be able to
use these 'signs'. SGML is one way of making such conventions.
>My mind is open, but so far SGML proponents have said nothing to make me
>change my mind. We all want document portability, but I don't think we'll
>ever see it from SGML. And if SGML isn't supposed to provide document
>portability, just what problem is it supposed to solve?
Document markup. I think SGML solves that, although perhaps not as
neatly as I would wish it did.
The interchange (or portability) part should be solved by the SDIF
standard. I haven't seen it, so I can't swear to it.
--
It would be interesting to hear from anyone with hands-on experience
with SGML products if they are capable of handling graphics, equations
and things like that, and how well they do.
Bill Bill Tuthill
>
> It would probably be a help to the readers of comp.text.sgml to have
Is there a newsgroup comp.text.sgml somewhere? Not where I work.
We investigated SGML two years ago, going so far as to send several people
to high-priced conferences. Our conclusion was that SGML doesn't solve
the main problem we need solved, which is document portability. Excuse the
baseball metaphor, but SGML has three strikes against it:
1. It originated at IBM, a company not renowned for software prowess.
2. The spec is bloated, bogus, pretentious, and incomprehensible.
3. It doesn't deal with graphics, tables, or equations (is that 5 strikes?).
It only provides text portability, which ASCII does more elegantly.
My mind is open, but so far SGML proponents have said nothing to make me
Bill Bill Tuthill
>
> ASCII isn't much of a help if I want to insert a Swedish { in the
> text. (That '{', of course, is an 'a' with an umlaut accent).
I probably should have said ISO 8859-1 (also known as ISO Latin-1),
not ASCII, but I didn't think many people would know what that is.
ISO Latin-1 is an 8-bit codeset identical to ASCII in the lower half,
but extended into the upper half with accent marks and characters
required in western Europe. Anders could produce any Scandinavian
character using this extended code set.
However, ISO Latin-1 doesn't solve the character encoding problem
outside western Europe. There are ISO standards for eastern Europe
(ISO 8859-2), Greece (ISO Greek), and Russia (ISO Cyrillic). There
are also standards for Japan (JLS) and elsewhere in Asia. But these
code sets are not mutually compatible; they are not interchangeable.
I believe the ultimate answer is Unicode, a 16-bit code set that
includes all known languages of the world in a single, interchangeable
code set. Developed by Joe Becker and others at Xerox, Apple, and
elsewhere, Unicode represents a tremendous leap forward. The main
reason 16 bits is sufficient is that Chinese, Japanese and Korean
pictographs have been combined so as to be complete and correctly
ordered, though not necessarily contiguous.
The main drawback to Unicode is that files will be twice as big. But
being able to exchange data without shifting and conversion is a huge
advantage. Space has even been left in the Unicode address space for
ancient writing systems such as hieroglyphics and cuneiform.
Edward Vielmetti
comp.text.sgml has just been created a few minutes ago, you should see
it soon. This should be the first message in it.
I hope to come up with a more reasonable introduction to the group in
the next few weeks. In the interim if you're on the internet the
site "sgml.math.lsa.umich.edu" has a directory /pub/sgml with some
stuff in it.
--Ed
Edward Vielmetti, U of Michigan math dept <e...@math.lsa.umich.edu>
Bob Stayton, Yoyodoc
>>
>We investigated SGML two years ago, going so far as to send several people
>to high-priced conferences. Our conclusion was that SGML doesn't solve
>the main problem we need solved, which is document portability. Excuse the
>baseball metaphor, but SGML has three strikes against it:
>
>1. It originated at IBM, a company not renowned for software prowess.
>
>2. The spec is bloated, bogus, pretentious, and incomprehensible.
>
>3. It doesn't deal with graphics, tables, or equations (is that 5 strikes?).
> It only provides text portability, which ASCII does more elegantly.
I won't touch your first two strikes, but the third requires
a response.
The most thorough implementation of SGML to date appears to
be CALS, the emerging Defense Department standard for
documentation. If you haven't looked at SGML for two
years, then you probably missed CALS.
CALS does address graphics, tables, and equations.
It sort of cheats on graphics by choosing three formats
from among the many, but hey, why invent another format?
The tables spec has gone through a major revision and
appears to address most of the needs of describing
complex tables. The equation part was adapted from ISO TR
9573 written by Anders Bergland.
The latest CALS spec also provides a method for identifying
a default format for each text element. That way you
can have your text and format it too |).
The baseline tag set of CALS provides actual tag names and
definitions that will become some kind of standard merely
by the weight of the DoD. Several vendors of publishing
software are providing CALS compliance, which will also help
make it a defacto standard for document interchange.
Of course, I don't know of anybody that is actually doing
this yet, but it is certainly feasible now.
bobs
Bob Stayton 425 Encinal Street
Technical Publications Santa Cruz, CA 95060
The Santa Cruz Operation, Inc. (408) 425-7222
...!uunet!sco!bobs
/* I don't speak for my company and they don't speak for me. */
Brian T. Schellenberger
|includes all known languages of the world in a single, interchangeable
|code set. Developed by Joe Becker and others at Xerox, Apple, and
|elsewhere, Unicode represents a tremendous leap forward. The main
|reason 16 bits is sufficient is that Chinese, Japanese and Korean
|pictographs have been combined so as to be complete and correctly
|ordered, though not necessarily contiguous.
|
|The main drawback to Unicode is that files will be twice as big. But
|being able to exchange data without shifting and conversion is a huge
|advantage. Space has even been left in the Unicode address space for
|ancient writing systems such as hieroglyphics and cuneiform.
This is be no means necessary. Even the "large" versions of Kanji and
such-like only have 6000 or so characters. Allowing for overlap, I would
immensely surprised if you couldn't take care of all the ideographic living
languages (mostly Chinese, Japanese, and Korean) in 15,000 characters, tops.
For non-ideograhpic languages, which tend to have no more than about 40
characters tops in no more than two variations (eg, capital and lowercase
for Roman; Katakana and Hiragana for Japanese), making for less than 100
characters total, we can accomidate more than 150 non-overlapping languages
in the same space. This leaves 2,000 slots for "special" symbols and
punctuation, while still coming in at less than 32,000 characters. Thus,
it can easily be fit into 15 bits. That way, we can have the 127 or so
most common chacters encoded into a 7 bits. Then if the eighth bit is
set, we know it starts a 15-bit character.
The obivous starting place it to make the 7-bit code be the current ASCII
code. I suspect that this is close to the set of most commonly used
characters world-wide, but it need not be the choice. If such a scheme is
used, most files will be only a little bit longer than they are now, and
(assuming ASCII is used as the base), computer code will increase not one
wit. Neither will English text. Since English is the lingua franca of the
world these days, this will includes a lot international text, and not just
English.
Finally, such a scheme is not all that difficult to work with. It is
certainly easier than shared 7- and 8-bit codes than include "shift"
characters.
--
-- Brian, the Man from Babble-on. b...@unx.sas.com
-- (Brian Schellenberger)
"And when the votes were cast, the winner was . . .
Mister James K. Polk, Napolean of the stump." -- THEY MIGHT BE GIANTS.
Bill Bill Tuthill
> |
> |The main drawback to Unicode is that files will be twice as big. But
> |being able to exchange data without shifting and conversion is a huge
> |advantage. Space has even been left in the Unicode address space for
> |ancient writing systems such as hieroglyphics and cuneiform.
>
> This is be no means necessary. Even the "large" versions of Kanji and
> such-like only have 6000 or so characters. Allowing for overlap, I would
> immensely surprised if you couldn't take care of all the ideographic living
> languages (mostly Chinese, Japanese, and Korean) in 15,000 characters, tops.
Wait a minute, do you mean it isn't necessary to combine Asian language
character sets? There are over 6000 Kanji characters in Japan, about the
same number in Korea and China (PRC) and around 14000 characters in Taiwan.
That adds up to at least 32000. Combinatory efforts done by the Unicode
people have reduced that total to around 20000.
Or do you mean 16 bits aren't necessary, 15 are enough? That means there
would only be 12767 empty slots after covering the Asian languages, which
is almost certainly not enough. I really don't think the world needs yet
another shift encoding algorithm, anyway.
Cheung Yukngo
b...@unx.sas.com (Brian T. Schellenberger) writes:
> |
> |The main drawback to Unicode is that files will be twice as big. But
> |being able to exchange data without shifting and conversion is a huge
> |advantage. Space has even been left in the Unicode address space for
> |ancient writing systems such as hieroglyphics and cuneiform.
>
> This is be no means necessary. Even the "large" versions of Kanji and
> such-like only have 6000 or so characters. Allowing for overlap, I would
> immensely surprised if you couldn't take care of all the ideographic living
> languages (mostly Chinese, Japanese, and Korean) in 15,000 characters, tops.
Wait a minute, do you mean it isn't necessary to combine Asian language
character sets? There are over 6000 Kanji characters in Japan, about the
same number in Korea and China (PRC) and around 14000 characters in Taiwan.
That adds up to at least 32000. Combinatory efforts done by the Unicode
people have reduced that total to around 20000.
Well, I don't think 6000 Chinese characters is a reasonable amount. It
is probably good enough to cover most of the Chinese surnames.
According to ``An Introduction to Chinese, Japanese and Korean
Computing'' by Huang and Huang, 74,000 is a reasonanle amount.
Granted, most of the characters are not used. But then you don't purge
a word just because it is not used in daily life---so the size of
Oxford English Dictionary.
I don't know anything about SGML. I hope SGML knows something about
Asian Languages.
Liam R. E. Quin
> world in a single, interchangeable code set.
b...@unx.sas.com (Brian T. Schellenberger) writes:
> [...] still coming in at less than 32,000 characters. Thus, it can easily
> be fit into 15 bits. [Then] the 127 most common chacters [could fit in] 7
> bits. Then if the eighth bit is set, we know it starts a 15-bit character.
It might make more sense to mandate that _all_ of the bytes of such an
extended character, except the last, have the top bit set. Then there is
no limit imposed on the number of characters, although of course some
software might croak on 48-bit characters :-) This also means that your
files aren't twice as big, or even four times as big, and you can still use
lots of glyphs.
This does mean that algorithms such as Boyer-Moore pattern matching have to
look at at most one extra byte per probe in some cases, to ensure that a
match isn't the last byte of a multi-byte character. With a shift, you'd
have to look at every byte in the input to remember the current mode.
With four-byte encodings you could have to look at up to three bytes. So
my scheme is no worse than a plain two-byte encoding in this way either.
You should also look at the work in progress by ISO committees such as
ISO/IEC JTC 1/SC 18/WG8 on Font Information Interchange (e.g. N1036), and
at ISO/IEC/DIS 9541-1. They're busily working on ways of transmitting font
and glyph information about the place...
Lee
--
Liam R. E. Quin, l...@sq.com, SoftQuad Inc., Toronto, +1 (416) 963-8337
/text/humour/quote: No such file or directory
Timo Lehtinen
Ok, where does one get a specfication of CALS ?
Timo
--
____/ ___ ___/ / Kivihaantie 8 C 25
/ / / SF-00310 HELSINKI, Finland
____ / / / Phone: +358 0 573 161, +358 49 424 012
Stream Technologies Inc. Fax: +358 0 571 384
Jim Howe
|>
|> Ok, where does one get a specfication of CALS ?
|>
|> Timo
You can get some information via FTP from DURER.CME.NIST.GOV in the /pub/cals
directory.
James W. Howe internet: j...@ifs.umich.edu
University of Michigan uucp: uunet!mailrus!ifs.umich.edu!jwh
Ann Arbor, MI 48103-4943