FTS-ки по структуре фидошных баз
Mithgol the Webmaster
* также было отослано в эхоконференцию RU.FTN.DEVELOP
* также было отослано в эхоконференцию SU.FIDOTECH
* Кстати, именно в SU.FidoTech или Ru.FTN.Develop такие вопросы лучше обсуждать
Так было 21:11 08 Feb 07 написано от Konstantin Kuzov к All:
KK> Hужен сабж, у меня были, но я их куда затерял, в инете с наскока не
KK> нашел... Интересует прежде всего JAM. Ради интереса попробовал перевести
KK> FGHI-Гейт на чистый PHP(с использованием классов Алекса Кочарина), по
KK> скорости на сквише даже быстрее, чем в скомпиленном бинарнике, а вот с
KK> джамом совсем всё плохо было... (на базе с 5000 сообщений под конец аж
KK> всего по 5 сообщений в секунду парсилось)... :-( Hемного переделал класс,
KK> стало по паре тысяч в секунду, но решение мне не особо нравится, хотелось
KK> бы понадежнее сделать...
FTSов по структурам фидошных баз не было никогда.
Что касается JAMовской базы, то по ней у меня есть такой документ:
======= Hачало цитаты из файла JAM-001.TXT =======
(http://bbs.nehybka.cz/mbseman/misc/jam.html)
Filename....: JAM-001
Rev.........: 001
Dated.......: 93-07-01
Status .....: Released
Subject.....: JAM message base proposal
Author......: Joaquim Homrighausen
Co-Authors..: Andrew Milner, Mats Birch, Mats Wallin
---------------------------------------------------------------------
JAM(mbp)
The Joaquim-Andrew-Mats Message Base Proposal
---------------------------------------------------------------------
Copyright 1993 Joaquim Homrighausen, Andrew Milner,
Mats Birch, Mats Wallin.
ALL RIGHTS RESERVED.
---------------------------------------------------------------------
=====================================================================
Restrictions
---------------------------------------------------------------------
JAM may be used by any developer as long as these specifications are
followed exactly. JAM may be used free-of-charge by any developer
for any purpose, commercially or otherwise.
This document may be freely copied and distributed, but must NEVER be
distributed in a modified form. If you have an enhancement request,
please contact the author of this document; do not change it
yourself.
All applications that support JAM must include one of the following
notices in their documentation and somewhere in the product's credit
section:
"JAM(mbp) - Copyright 1993 Joaquim Homrighausen, Andrew Milner,
Mats Birch, Mats Wallin.
ALL RIGHTS RESERVED."
or
"This product uses the JAM(mbp) API -
Copyright 1993 Joaquim Homrighausen, Andrew Milner, Mats Birch,
Mats Wallin. ALL RIGHTS RESERVED."
No organization, company, person, entity, or other being may impose
any fees for any reason for providing this document or the
accompanying API. This document and the accompanying API may not be
sold or otherwise transferred for personal or company gain under any
circumstances.
=====================================================================
Definitions and general notes
---------------------------------------------------------------------
CURRENTREV 1
JAM The JAM message base format.
CRC Cyclic Redundancy Check. All CRC values
calculated on strings must assume that the
data within the string has been converted
to lowercase (A-Z = a-z).
CRC-32 32-bit CRC (as used in the Zmodem file
transfer protocol) value. The polynom for
a CRC-32 is edb88320H and the CRC-32 seed
is -1L (ffffffffH).
uchar Unsigned 8-bit value
ushort Unsigned 16-bit value
ulong Unsigned 32-bit value
UNIX date An ulong representing the number of seconds
since midnight, January 1, 1970. UNIX-style
dates is the only form of time stamps used
in JAM (1).
Message # The physical record number within the index
file is used as a message number. The
lowest message number is one (1) and the
highest message number is 4294967295
(ffffffffH).
FTN FidoNet Technology Network
FTS FidoNet Technical Standard
(1) All timestamps created locally (i.e. those not imported from
other systems) are stored in local time.
=====================================================================
Files
---------------------------------------------------------------------
Each conference is made up from four files. How and where these files
are stored and named is implementation dependant. The only file with
a fixed minimum size is the .JHR (header data) file. It has a 1024-
byte block used to hold information about a specific message area as
described later.
filename.JHR - Message header data
filename.JDT - Message text data
filename.JDX - Message index
filename.JLR - Lastread information
A future revision of JAM may also include a file that holds the
following three items:
- The highest assigned user number
- The last generated message ID
- A global conference list with the conference name, description,
and physical location of the message base.
=====================================================================
.JHR file header
---------------------------------------------------------------------
Below is the format of the 1024-byte record at the beginning of all
.JHR files. The first actual message header starts at offset 1024 in
the .JHR file.
FixedHeaderInfoStruct:
ulong Signature; // <J><A><M> followed by <NUL>
ulong datecreated; // Creation date
ulong modcounter; // Update counter
ulong activemsgs; // Number of active (not deleted) msgs
ulong passwordcrc; // CRC-32 of password to access
ulong basemsgnum; // Lowest message number in index file
uchar RESERVED[1000]; // Reserved space
end;
MODCOUNTER must be incremented and updated on disk each time an
application modifies the contents of the message base. When it
reaches ffffffffH, it wraps to zero.
---------------------------------------------------------------------
BaseMsgNum Lowest message number in index file
---------------------------------------------------------------------
This field determines the lowest message number in the index file.
The value for this field is one (1) when a message area is first
created. By using this field, a message area can be packed (deleted
messages are removed) without renumbering it. If BaseMsgNum contains
500, the first index record points to message number 500.
BaseMsgNum has to be taken into account when an application
calculates the next available message number (for creating new
messages) as well as the highest and lowest message number in a
message area.
---------------------------------------------------------------------
????????.JHR Message headers
---------------------------------------------------------------------
The .JHR file contains none or more Header records. Each record
define one message and contains information about the message and its
text (if any). The Header record is of variable length. The layout of
the Header record follows.
MessageHeader:
MessageFixedHeader:
ulong Signature; // <J><A><M> followed by <NUL>
ushort Revision; // Revision level of header (1)
ushort ReservedWord; // Reserved for future use
ulong SubfieldLen; // Length of subfields (2)
ulong TimesRead; // Number of times message read
ulong MSGIDcrc; // CRC-32 of MSGID line (3)
ulong REPLYcrc; // CRC-32 of REPLY line (3)
ulong ReplyTo; // This msg is a reply to..
ulong Reply1st; // First reply to this msg
ulong Replynext; // Next msg in reply chain
ulong DateWritten; // When msg was written
ulong DateReceived; // When msg was read by recipient
ulong DateProcessed;// When msg was processed by tosser/
// scanner
ulong MessageNumber;// Message number (1-based)
ulong Attribute; // Msg attribute, see "Msg Attributes"
ulong Attribute2; // Reserved for future use
ulong Offset; // Offset of text in ????????.JDT file
ulong TxtLen; // Length of message text
ulong PasswordCRC; // CRC-32 of password to access message
ulong Cost; // Cost of message
end;
SubField1 // Extra fields as defined below
.
.
SubFieldXX
end;
(1) This field is intended for future revisions of the specifications
to allow the use of a different fixed-length binary message
header. The current revision level is one (1).
(2) The SubfieldLen field is set to zero (0) if the header does not
have any subfield data. I.e. the length of the binary header is
not included in this field.
(3) When calculating the CRC-32 of the MSGID and REPLY lines, the
text ^aMSGID: and ^aREPLY: should be removed as well as all
leading and trailing white space characters.
The SubField structure is made up of an ID, a length specifier, and
a block of data. Zero or more subfields may follow the fixed-length
binary header. SubFields are not stored in any specific order and
are not terminated by any specific character unless otherwise
specified.
SubField:
ushort LoID; // Field ID, 0-65535
ushort HiID; // Reserved for future use
ulong datlen; // Length of buffer that follows
uchar Buffer[datlen]; // DATLEN bytes of data
end;
---------------------------------------------------------------------
Defined LoID codes
---------------------------------------------------------------------
ID=0, Name=OADDRESS
A network address. This is used to specify the originating address.
More than one OADDRESS field may exist. DATLEN must not exceed 100
characters. For a FidoNet-style address, this field must follow the
ZONE:NET/NODE.POINT@DOMAIN format where .POINT is excluded if zero
and @DOMAIN is excluded if unknown.
ID=1, Name=DADDRESS
A network address. This is used to specify the destination address.
More than one DADDRESS field may exist (e.g. carbon copies). DATLEN
must not exceed 100 characters. For a FidoNet-style address, this
field must follow the ZONE:NET/NODE.POINT@DOMAIN format where .POINT
is excluded if zero and @DOMAIN is excluded if unknown.
ID=2, Name=SENDERNAME
The sender (author) of the message. DATLEN must not exceed 100
characters.
ID=3, Name=RECEIVERNAME
The recipient of the message. DATLEN must not exceed 100 characters.
ID=4, Name=MSGID
Used to store the message identification data. All data not relevant
to the actual ID string, including leading and trailing white space
characters should be removed. DATLEN must not exceed 100 characters.
ID=5, Name=REPLYID
Used to store the message reply data. All data not relevant to the
actual reply string, including leading and trailing white space
characters should be removed. DATLEN must not exceed 100 characters.
ID=6, Name=SUBJECT
The subject of the message. DATLEN must not exceed 100 characters.
Note that this field may not be used for FidoNet-style file attaches
or file requests.
ID=7, Name=PID
Used to store the FTN PID kludge line. Only the actual PID data is
stored and ^aPID: is stripped along with any leading and trailing
white space characters. DATLEN must not exceed 40 characters.
ID=8, Name=TRACE
This is also referred to as ^aVia information in FTNs. It contains
information about a system which the message has travelled through.
The format of the field is <YYYYMMDDHHMMSS><Network address> where:
YYYY is the year (1992-9999)
MM is the month (01-12)
DD is the day (01-31)
HH is the hour (00-23)
MM is the minute (00-59)
SS is the second (00-59)
The timestamp is stored in ASCII (0-9) characters. The network
address is the address of the system. It is expressed in ASCII
notation in the native format of the forwarding system.
ID=9, Name=ENCLOSEDFILE
A file attached to the message. Only one filename may be specified
per subfield. No wildcard characters are allowed. If this subfield
is present in a message header, the ATTRIBUTE must include the
MSG_FILEATTACH bit.
ID=10, Name=ENCLOSEDFILEWALIAS
Identical to ENCLOSEDFILE with the exception that the filename is
followed by a <NUL> (00H) and an alias filename to be transmited to
the remote system in place of the local name of the file.
ID=11, Name=ENCLOSEDFREQ
A request for one or more files. Only one filemask may be specified
per subfield. If the filemask contains a complete path, it is to be
regarded as an update file request. If this subfield is present in a
message header, the ATTRIBUTE must include the MSG_FILEREQUEST bit.
To indicate that a password is to be transmitted along with the
request, a <NUL> (00H) character followed by the password is
appended. E.g. SECRET*.*<NUL>MYPASSWORD.
ID=12, Name=ENCLOSEDFILEWCARD
One or more files attached to the message. Only one filename may be
specified per subfield. Wildcard characters are allowed. If this
subfield is present in a message header, the ATTRIBUTE must include
the MSG_FILEATTACH bit.
ID=13, Name=ENCLOSEDINDIRECTFILE
One or more files attached to the message. The filename points to an
ASCII file with one filename entry per line. If alias filenames are
to be used, they are specified after the actual filename and
separated by a <NUL> (00H) character, e.g. C:\MYFILE.LZH<NUL>NEWS.
Wildcard characters are not allowed.
ID=1000, Name=EMBINDAT
Reserved for future use.
ID=2000, Name=FTSKLUDGE
An FTS-compliant "kludge" line not otherwise represented here. All
data not relevant to the actual kludge line, including leading and
trailing white space and ^A (01H) characters should be removed.
DATLEN must not exceed 255 characters. The FTS kludges INTL, TOPT,
and FMPT must never be stored as separate SubFields. Their data must
be extracted and used for the address SubFields.
ID=2001, Name=SEENBY2D
Used to store two-dimensional (net/node) SEEN-BY information often
used in FTN conference environments. Only the actual SEEN-BY data is
stored and ^aSEEN-BY: or SEEN-BY: is stripped along with any leading
and trailing white space characters.
ID=2002, Name=PATH2D
Used to store two-dimensional (net/node) PATH information often used
in FTN conference environments. Only the actual PATH data is stored
and ^aPATH: is stripped along with any leading and trailing white
space characters.
ID=2003, Name=FLAGS
Used to store the FTN FLAGS kludge information. Note that all FLAG
options that have binary representation in the JAM message header
must be removed from the FLAGS string prior to storing it. Only
the actual flags option string is stored and ^aFLAGS is stripped
along with any leading and trailing white space characters.
ID=2004, Name=TZUTCINFO
Time zone information. This subfield consists of four mandatory
bytes and one optional. The first character may be a plus (+) or a
minus (-) character to indicate a location east (plus) or west
(minus) of UTC 0000. The plus character is implied unless the first
character is a minus character. The following four bytes must be
digits in the range zero through nine and indicates the offset in
hours and minutes. E.g. 0100 indicates an offset of one hour east of
UTC.
---------------------------------------------------------------------
Message attributes
---------------------------------------------------------------------
MSG_LOCAL (0x00000001L) // Msg created locally
MSG_INTRANSIT (0x00000002L) // Msg is in-transit
MSG_PRIVATE (0x00000004L) // Private
MSG_READ (0x00000008L) // Read by addressee
MSG_SENT (0x00000010L) // Sent to remote
MSG_KILLSENT (0x00000020L) // Kill when sent
MSG_ARCHIVESENT (0x00000040L) // Archive when sent
MSG_HOLD (0x00000080L) // Hold for pick-up
MSG_CRASH (0x00000100L) // Crash
MSG_IMMEDIATE (0x00000200L) // Send Msg now, ignore restrictions
MSG_DIRECT (0x00000400L) // Send directly to destination
MSG_GATE (0x00000800L) // Send via gateway
MSG_FILEREQUEST (0x00001000L) // File request
MSG_FILEATTACH (0x00002000L) // File(s) attached to Msg
MSG_TRUNCFILE (0x00004000L) // Truncate file(s) when sent
MSG_KILLFILE (0x00008000L) // Delete file(s) when sent
MSG_RECEIPTREQ (0x00010000L) // Return receipt requested
MSG_CONFIRMREQ (0x00020000L) // Confirmation receipt requested
MSG_ORPHAN (0x00040000L) // Unknown destination
MSG_ENCRYPT (0x00080000L) // Msg text is encrypted (1)
MSG_COMPRESS (0x00100000L) // Msg text is compressed (1)
MSG_ESCAPED (0x00200000L) // Msg text is seven bit ASCII (1)
MSG_FPU (0x00400000L) // Force pickup
MSG_TYPELOCAL (0x00800000L) // Msg is for local use only
MSG_TYPEECHO (0x01000000L) // Msg is for conference distribution
MSG_TYPENET (0x02000000L) // Msg is direct network mail
MSG_NODISP (0x20000000L) // Msg may not be displayed to user
MSG_LOCKED (0x40000000L) // Msg is locked, no editing possible
MSG_DELETED (0x80000000L) // Msg is deleted
(1) This revision of JAM does not include compression, encryption, or
escaping. The bits are reserved for future use.
=====================================================================
????????.JDT Message text
---------------------------------------------------------------------
The .JDT file contains the text of messages. The text is stored as an
stream of seven or eight bit ASCII data. Allowed characters in the
text are 00H through ffH unless the header ATTRIBUTE field has the
MSG_ESCAPED bit enabled, in which case the legal range of data is 20H
through 7eH.
An escaped character is stored as \<hex> where <hex> is the two digit
hexadecimal ASCII value of the character. A single \ is stored as \\
or \5C. The case of the hexadecimal ASCII value is irrelevant, i.e.
5c is treated as 5C.
=====================================================================
????????.JDX Message index
---------------------------------------------------------------------
The .JDX file is used to quickly locate messages for any given user
name or to locate a message with a specific number. Each record in
the file consists of two ulongs. The first ulong holds the CRC-32 of
the recipient's name (lowercase), the second ulong holds the
physical offset of the message header in the .JHR (header) file.
The record number (+BaseMsgNum) within the .JDX file determines a
message's number.
If both ulongs are -1 (ffffffffH), there is no corresponding message
header.
=====================================================================
????????.JLR Lastread storage
---------------------------------------------------------------------
The .JLR file is used to maintain a user's position within a message
area. The layout of the "lastread" record follows. One record per
user is required.
LastRead:
ulong UserCRC; // CRC-32 of user name (lowercase) (1)
ulong UserID; // Unique UserID
ulong LastReadMsg; // Last read message number
ulong HighReadMsg; // Highest read message number
end;
(1) The functions to convert a string to lowercase characters that
are provided in the API will only convert characters A-Z (into
a-z). It is required that this convention is followed by all
applications.
The UserID field is a unique number for each user. If the "lastread"
record is deleted, UserCRC and UserID are both set to -1
(ffffffffH). An application may not depend on any specific order in
the .JLR file. A user's "lastread" record may appear anywhere in the
file and must be searched for when retrieving it and when storing an
updated record.
=====================================================================
Updating message headers
---------------------------------------------------------------------
If a header record grows after is has been retrieved from the .JHR
file, it must be appended to the end of the .JHR file since it would
overwrite the following header otherwise. The .JDX file must be
properly updated to indicate the new location of the header record.
The old header record must be changed to indicate that it has been
deleted by setting the MSG_DELETED bit in the Attribute field and the
TextLen field to zero (to prevent a maintenance program from removing
the message text that is now pointed to by another header).
=====================================================================
Message base sharing and locking
---------------------------------------------------------------------
To allow several programs to access the message base at any given
time, region locking is used to protect the message base from being
corrupted during updates.
When an application needs to write to any of the message base files,
it must first attempt to lock the first byte of the .JHR (header)
file. If the lock call fails, the application must either fail or
attempt to lock the file again. The message base files may under no
circumstances be updated if the application cannot successfully lock
the .JHR file.
Note that data acquired (read) from the message base may not be used
when writing data to the message base, unless the application has
maintained a lock of the message base from the time the data was
acquired or the MODCOUNTER field is the same as when the data was
acquired.
The application must open the files in shareable (DENYNONE) read/
write or readonly mode. The only exception to this is an application
that requires exclusive access to the message base, such as a message
base maintenance utility, it should open the files in non-shareable
(DENYALL) read/write mode.
=====================================================================
Reply threads and linking
---------------------------------------------------------------------
JAM introduces a new reply link pointer, not commonly used today.
This section is an attempt to describe how reply threads, reply
linking, and this new reply link pointer is implemented in JAM.
One of the major differences is that reply threads in JAM are not
based on similar or identical subjects of messages since this method
does not allow for proper reply threads.
The method used in JAM is based on the immediate relation between any
given message and direct replies to it. This is supported by many
message editors by using the MSGID and REPLY FTS kludge fields. These
are common, although expressed differently, in messages not based on
FidoNet technology, such as RFC-822. The obvious advantages include
allowing a program to easily find the original message to a reply,
and to find all replies to any given message.
The reply thread information consists of three fields: ReplyTo,
Reply1st, and ReplyNext. The reason for three fields, as opposed to
just two, is that with two fields, it is only possible to keep track
of the original message of a reply (which is sufficient) and one
reply to any given message (which is not sufficient). With three
fields, it is possible to maintain a thread of any number of replies
to any given message.
In the description of the different fields below, the following
messages and message numbers will be referred to:
1 -> 2 -> 4 -> 5
: :
: +--> 8
:
+--> 3 -> 7
:
+--> 6
Message number two, three, and six are replies to message number one.
Message number four and eight are replies to message number two.
Message number seven is a reply to message number three.
Message number five is a reply to message number four.
---------------------------------------------------------------------
ReplyTo
---------------------------------------------------------------------
This field holds the number of the message that this message is a
reply to. In the example above, the ReplyTo field would contain the
following values:
Message number one would contain zero; message number two, three, and
six, would contain one; message number four and eight would contain
two; message number seven would contain three, and message number
five would contain four.
---------------------------------------------------------------------
Reply1st
---------------------------------------------------------------------
This field holds the number of the first message that is a reply to
this message. In the example above, the Reply1st field would contain
the following values:
Message number one would contain two, message number three would
contain seven, and message number four would contain five. All other
messages would contain zero.
---------------------------------------------------------------------
ReplyNext
---------------------------------------------------------------------
This field is used to create the actual message thread or chain. In
the event that there is more than one reply to any given message, it
is necessary to maintain a thread of all the replies; this is due to
the fact that the original message can only hold information about
the first reply (the Reply1st field) to it.
The first reply (which the original message's Reply1st field holds),
has its ReplyNext field pointing to the second reply, the second
reply's ReplyNext field poinst to the third reply, and so on.
In the example above, the ReplyNext field would contain the following
values:
Message number two would contain three, message number three would
contain six, and message number four would contain eight. All other
messages would contain zero.
=====================================================================
Contacts
---------------------------------------------------------------------
Joaquim Homrighausen Telefax: +352 316 702
389, route d'Arlon Modem: +352 316 702
L-8011 Strassen eMail: 2:270/17@fidonet
Luxembourg jo...@abs.lu
Andrew Milner Telefax: +352 251 621
9a, Boulevard Joseph II Modem: +352 251 621
L-1840 Belair eMail: 2:270/18@fidonet
Luxembourg and...@fido.lu
Mats Wallin Telefax: +46 8 6453285
F"rskottsv"gen 11 Modem: +46 8 6453882
S-126 44 H"gersten eMail: 2:201/329@fidonet
Sweden m...@fido.lu
======= Конец цитаты из файла JAM-001.TXT =======
Ceterum censeo Diptaunum esse delendam! [Ru.Mozilla] http://Mithgol.Ru/
Mithgol the Webmaster. [Team В эхолисте ,,ё`` уместим] [Team А я меняю subj]
... Венец ── деве конец. (из чужих ориджинов)
Alex Kocharin
MtW> FTSов по структурам фидошных баз не было никогда.
FTS-0001.
`-._ --- Alexander Kocharin ---
Vadim Goncharov
On Sat, 10 Feb 2007 20:19:14 +0300; Alex Kocharin wrote about 'FTS-ки по структуре фидошных баз':
MtW>> FTSов по структурам фидошных баз не было никогда.
AK> FTS-0001.
Это документ в основном посвящен протоколам, и совершенно не запрещает
иметь другие базы на 7 уровне. Что в действительности и наблюдаем
- его форматом никто не пользуется.
--
WBR, Vadim Goncharov. ICQ#166852181 mailto:vadim_n...@mail.ru
[Moderator of RU.ANTI-ECOLOGY][FreeBSD][http://antigreen.org][LJ:/nuclight]
Alex Kocharin
MtW>>> FTSов по структурам фидошных баз не было никогда.
AK>> FTS-0001.
VG> Это документ в основном посвящен протоколам, и совершенно не запрещает
VG> иметь другие базы на 7 уровне. Что в действительности и наблюдаем
VG> - его форматом никто не пользуется.
Этот FTS предлагает один из возможных форматов (aka "структур") фидошных баз.
Что опровергает утверждение MtW...
Vadim Goncharov
On Sun, 11 Feb 2007 21:04:00 +0300; Alex Kocharin wrote about 'FTS-ки по структуре фидошных баз':
MtW>>>> FTSов по структурам фидошных баз не было никогда.
AK>>> FTS-0001.
VG>> Это документ в основном посвящен протоколам, и совершенно не запрещает
VG>> иметь другие базы на 7 уровне. Что в действительности и наблюдаем
VG>> - его форматом никто не пользуется.
AK> Этот FTS предлагает один из возможных форматов (aka "структур") фидошных баз.
AK> Что опровергает утверждение MtW...
Отнюдь. MtW здесь формально прав, потому что не было FTS конкретно по
формату баз. FTS-1 есть основной стандарт, посвящен много чему, базы там
до кучи.
Alex Kocharin
MtW>>>>> FTSов по структурам фидошных баз не было никогда.
AK>>>> FTS-0001.
VG>>> Это документ в основном посвящен протоколам, и совершенно не запрещает
VG>>> иметь другие базы на 7 уровне. Что в действительности и наблюдаем
VG>>> - его форматом никто не пользуется.
AK>> Этот FTS предлагает один из возможных форматов (aka "структур") фидошных
AK>> баз. Что опровергает утверждение MtW...
VG> Отнюдь. MtW здесь формально прав, потому что не было FTS конкретно по
VG> формату баз. FTS-1 есть основной стандарт, посвящен много чему, базы
VG> там до кучи.
Ну _только_ по структурам - да, не было. А FTS, в котором содержится описание
базы (что больше подходит по смыслу к исходному письму) был - адын штука.