{Proposal} Standard Packet Formats

[stanl...@gmail.com]

I don't know how many of you have looked at the Discretion settings,
but basically 90% of the packets and their fields are defined in a
configurable text file which is then loaded in. I think this is much
better than the current practice of reverse engineering or looking at
different incomplete packet lists. It would also make writing a new
client/chatbot/server that much easier. Here's what the Discretion one
looks like:

[Packet Templates]

encryption request type = 0x01
encryption request iscore = 1
encryption request field count = 2
encryption request field 0 name = key
encryption request field 0 type = int
encryption request field 0 length = 4
encryption request field 1 name = protocol
encryption request field 1 type = int
encryption request field 1 length = 2

encryption response type = 0x02
encryption response iscore = 1
encryption response field count = 1
encryption response field 0 name = key
encryption response field 0 type = int
encryption response field 0 length = 4

reliable response type = 0x04
reliable response iscore = 1
reliable response field count = 1
reliable response field 0 name = id
reliable response field 0 type = int
reliable response field 0 length = 4

disconnect type = 0x07
disconnect iscore = 1
disconnect field count = 0

password request type = 0x09
password request field count = 12
password request field 0 name = new user
password request field 0 type = int
password request field 0 length = 1
password request field 1 name = name
password request field 1 type = string
password request field 1 length = 32
password request field 2 name = password
password request field 2 type = string
password request field 2 length = 32
password request field 3 name = machine id
password request field 3 type = int
password request field 3 length = 4
password request field 4 name = connect type
password request field 4 type = int
password request field 4 length = 1
password request field 5 name = timezone bias
password request field 5 type = int
password request field 5 length = 2
password request field 6 name = unknown1
password request field 6 type = int
password request field 6 length = 2
password request field 7 name = client version
password request field 7 type = int
password request field 7 length = 2
password request field 8 name = mem checksum1
password request field 8 type = int
password request field 8 length = 4
password request field 9 name = mem checksum2
password request field 9 type = int
password request field 9 length = 4
password request field 10 name = permission id
password request field 10 type = int
password request field 10 length = 4
password request field 11 name = unknown2
password request field 11 type = string
password request field 11 length = 12

password response type = 0x0A
password response field count = 10
password response field 0 name = login response
password response field 0 type = int
password response field 0 length = 1
password response field 1 name = server version
password response field 1 type = int
password response field 1 length = 4
password response field 2 name = unknown1
password response field 2 type = int
password response field 2 length = 4
password response field 3 name = subspace.exe checksum
password response field 3 type = int
password response field 3 length = 4
password response field 4 name = unknown2
password response field 4 type = int
password response field 4 length = 4
password response field 5 name = unknown3
password response field 5 type = int
password response field 5 length = 1
password response field 6 name = registration request
password response field 6 type = int
password response field 6 length = 1
password response field 7 name = code checksum
password response field 7 type = int
password response field 7 length = 4
password response field 8 name = news checksum
password response field 8 type = int
password response field 8 length = 4
password response field 9 name = unknown4
password response field 9 type = string
password response field 9 length = 8

keep alive type = 0x27
keep alive field count = 0

sync request type = 0x18
sync request field count = 4
sync request field 0 name = prize seed
sync request field 0 type = int
sync request field 0 length = 4
sync request field 1 name = door seed
sync request field 1 type = int
sync request field 1 length = 4
sync request field 2 name = timestamp
sync request field 2 type = int
sync request field 2 length = 4
sync request field 3 name = checksum key
sync request field 3 type = int
sync request field 3 length = 4

sync ping type = 0x05
sync ping iscore = 1
sync ping field count = 3
sync ping field 0 name = timestamp
sync ping field 0 type = int
sync ping field 0 length = 4
sync ping field 1 name = packets sent
sync ping field 1 type = int
sync ping field 1 length = 4
sync ping field 2 name = packets received
sync ping field 2 type = int
sync ping field 2 length = 4

sync pong type = 0x06
sync pong iscore = 1
sync pong field count = 2
sync pong field 0 name = original timestamp
sync pong field 0 type = int
sync pong field 0 length = 4
sync pong field 1 name = server timestamp
sync pong field 1 type = int
sync pong field 1 length = 4

cancel stream request type = 0x0b
cancel stream request iscore = 1
cancel stream request field count = 0

cancel stream response type = 0x0c
cancel stream response iscore = 1
cancel stream response field count = 0

;;; we define this guy because we don't get packet information until
after the login sequece,
;;; whereas a disconnect message could happen before then
incoming chat type = 0x07
incoming chat field count = 4
incoming chat field 0 name = type
incoming chat field 0 type = int
incoming chat field 0 length = 1
incoming chat field 1 name = sound
incoming chat field 1 type = int
incoming chat field 1 length = 1
incoming chat field 2 name = pid
incoming chat field 2 type = int
incoming chat field 2 length = 2
incoming chat field 3 name = message
incoming chat field 3 type = null terminated string

file data type = 0xd0
file data field count = 3
file data field 0 name = crc32
file data field 0 type = int
file data field 0 length = 4
file data field 1 name = length
file data field 1 type = int
file data field 1 length = 4
file data field 2 name = http path
file data field 2 type = string
file data field 2 length = 128

end file list type = 0xd1
end file list field count = 0

; Gameplayish below

change ship request type = 0x18
change ship request field count = 1
change ship request field 0 name = ship
change ship request field 0 type = int
change ship request field 0 length = 1

arena login type = 0x01
arena login field count = 7
arena login field 0 name = ship
arena login field 0 type = int
arena login field 0 length = 1
arena login field 1 name = allow audio
arena login field 1 type = int
arena login field 1 length = 2
arena login field 2 name = x resolution
arena login field 2 type = int
arena login field 2 length = 2
arena login field 3 name = y resolution
arena login field 3 type = int
arena login field 3 length = 2
arena login field 4 name = arena number
arena login field 4 type = int
arena login field 4 length = 2
arena login field 5 name = arena name
arena login field 5 type = string
arena login field 5 length = 16
arena login field 6 name = lvz
arena login field 6 type = int
arena login field 6 length = 1

player entering type = 0x03
player entering field count = 13
player entering field 0 name = ship
player entering field 0 type = int
player entering field 0 length = 1
player entering field 1 name = unknown
player entering field 1 type = int
player entering field 1 length = 1
player entering field 2 name = name
player entering field 2 type = string
player entering field 2 length = 20
player entering field 3 name = squad
player entering field 3 type = string
player entering field 3 length = 20
player entering field 4 name = flag points
player entering field 4 type = int
player entering field 4 length = 4
player entering field 5 name = kill points
player entering field 5 type = int
player entering field 5 length = 4
player entering field 6 name = pid
player entering field 6 type = int
player entering field 6 length = 2
player entering field 7 name = freq
player entering field 7 type = int
player entering field 7 length = 2
player entering field 8 name = kills
player entering field 8 type = int
player entering field 8 length = 2
player entering field 9 name = deaths
player entering field 9 type = int
player entering field 9 length = 2
player entering field 10 name = turret pid
player entering field 10 type = int
player entering field 10 length = 2
player entering field 11 name = flags
player entering field 11 type = int
player entering field 11 length = 2
player entering field 12 name = koth
player entering field 12 type = int
player entering field 12 length = 1


player leaving type = 0x04
player leaving field count = 1
player leaving field 0 name = pid
player leaving field 0 type = int
player leaving field 0 length = 2

outgoing chat type = 0x06
outgoing chat field count = 4
outgoing chat field 0 name = type
outgoing chat field 0 type = int
outgoing chat field 0 length = 1
outgoing chat field 1 name = sound
outgoing chat field 1 type = int
outgoing chat field 1 length = 1
outgoing chat field 2 name = pid
outgoing chat field 2 type = int
outgoing chat field 2 length = 2
outgoing chat field 3 name = message
outgoing chat field 3 type = null terminated string

freq change type = 0x0d
freq change field count = 3
freq change field 0 name = pid
freq change field 0 type = int
freq change field 0 length = 2
freq change field 1 name = freq
freq change field 1 type = int
freq change field 1 length = 2
freq change field 2 name = unknown
freq change field 2 type = int
freq change field 2 length = 1

pid change type = 0x01
pid change field count = 1
pid change field 0 name = pid
pid change field 0 type = int
pid change field 0 length = 2

incoming file transfer type = 0x10
incoming file transfer field count = 2
incoming file transfer field 0 name = file name
incoming file transfer field 0 type = string
incoming file transfer field 0 length = 16
incoming file transfer field 1 name = data
incoming file transfer field 1 type = raw

stp request type = 0x30
stp request field count = 1
stp request field 0 name = file name
stp request field 0 type = string
stp request field 0 length = 16

freq ship changed type = 0x1d
freq ship changed field count = 3
freq ship changed field 0 name = ship
freq ship changed field 0 type = int
freq ship changed field 0 length = 1
freq ship changed field 1 name = pid
freq ship changed field 1 type = int
freq ship changed field 1 length = 2
freq ship changed field 2 name = freq
freq ship changed field 2 type = int
freq ship changed field 2 length = 2

c2s position type = 0x03
c2s position field count = 11
c2s position field 0 name = direction
c2s position field 0 type = int
c2s position field 0 length = 1
c2s position field 1 name = timestamp
c2s position field 1 type = int
c2s position field 1 length = 4
c2s position field 2 name = xvel
c2s position field 2 type = int
c2s position field 2 length = 2
c2s position field 3 name = ypos
c2s position field 3 type = int
c2s position field 3 length = 2
c2s position field 4 name = checksum
c2s position field 4 type = int
c2s position field 4 length = 1
c2s position field 5 name = togglables
c2s position field 5 type = int
c2s position field 5 length = 1
c2s position field 6 name = xpos
c2s position field 6 type = int
c2s position field 6 length = 2
c2s position field 7 name = yvel
c2s position field 7 type = int
c2s position field 7 length = 2
c2s position field 8 name = bounty
c2s position field 8 type = int
c2s position field 8 length = 2
c2s position field 9 name = energy
c2s position field 9 type = int
c2s position field 9 length = 2
c2s position field 10 name = weapon info
c2s position field 10 type = int
c2s position field 10 length = 2

s2c weapon with energy type = 0x05
s2c weapon with energy field count = 13
s2c weapon with energy field 0 name = direction
s2c weapon with energy field 0 type = int
s2c weapon with energy field 0 length = 1
s2c weapon with energy field 1 name = timestamp
s2c weapon with energy field 1 type = int
s2c weapon with energy field 1 length = 2
s2c weapon with energy field 2 name = xpos
s2c weapon with energy field 2 type = int
s2c weapon with energy field 2 length = 2
s2c weapon with energy field 3 name = yvel
s2c weapon with energy field 3 type = int
s2c weapon with energy field 3 length = 2
s2c weapon with energy field 4 name = pid
s2c weapon with energy field 4 type = int
s2c weapon with energy field 4 length = 2
s2c weapon with energy field 5 name = xvel
s2c weapon with energy field 5 type = int
s2c weapon with energy field 5 length = 2
s2c weapon with energy field 6 name = checksum
s2c weapon with energy field 6 type = int
s2c weapon with energy field 6 length = 1
s2c weapon with energy field 7 name = toggleables
s2c weapon with energy field 7 type = int
s2c weapon with energy field 7 length = 1
s2c weapon with energy field 8 name = ping
s2c weapon with energy field 8 type = int
s2c weapon with energy field 8 length = 1
s2c weapon with energy field 9 name = ypos
s2c weapon with energy field 9 type = int
s2c weapon with energy field 9 length = 2
s2c weapon with energy field 10 name = bounty
s2c weapon with energy field 10 type = int
s2c weapon with energy field 10 length = 2
s2c weapon with energy field 11 name = weapon
s2c weapon with energy field 11 type = int
s2c weapon with energy field 11 length = 2
s2c weapon with energy field 12 name = energy
s2c weapon with energy field 12 type = int
s2c weapon with energy field 12 length = 2

s2c weapon type = 0x05
s2c weapon field count = 12
s2c weapon field 0 name = direction
s2c weapon field 0 type = int
s2c weapon field 0 length = 1
s2c weapon field 1 name = timestamp
s2c weapon field 1 type = int
s2c weapon field 1 length = 2
s2c weapon field 2 name = xpos
s2c weapon field 2 type = int
s2c weapon field 2 length = 2
s2c weapon field 3 name = yvel
s2c weapon field 3 type = int
s2c weapon field 3 length = 2
s2c weapon field 4 name = pid
s2c weapon field 4 type = int
s2c weapon field 4 length = 2
s2c weapon field 5 name = xvel
s2c weapon field 5 type = int
s2c weapon field 5 length = 2
s2c weapon field 6 name = checksum
s2c weapon field 6 type = int
s2c weapon field 6 length = 1
s2c weapon field 7 name = toggleables
s2c weapon field 7 type = int
s2c weapon field 7 length = 1
s2c weapon field 8 name = ping
s2c weapon field 8 type = int
s2c weapon field 8 length = 1
s2c weapon field 9 name = ypos
s2c weapon field 9 type = int
s2c weapon field 9 length = 2
s2c weapon field 10 name = bounty
s2c weapon field 10 type = int
s2c weapon field 10 length = 2
s2c weapon field 11 name = weapon
s2c weapon field 11 type = int
s2c weapon field 11 length = 2

; you are now in the game!
in game type = 0x02
in game field count = 0

small position type = 0x28
small position field count = 10
small position field 0 name = direction
small position field 0 type = int
small position field 0 length = 1
small position field 1 name = timestamp
small position field 1 type = int
small position field 1 length = 2
small position field 2 name = xpos
small position field 2 type = int
small position field 2 length = 2
small position field 3 name = c2sping
small position field 3 type = int
small position field 3 length = 1
small position field 4 name = bounty
small position field 4 type = int
small position field 4 length = 1
small position field 5 name = pid
small position field 5 type = int
small position field 5 length = 1
small position field 6 name = togglables
small position field 6 type = int
small position field 6 length = 1
small position field 7 name = yvel
small position field 7 type = int
small position field 7 length = 2
small position field 8 name = ypos
small position field 8 type = int
small position field 8 length = 2
small position field 9 name = xvel
small position field 9 type = int
small position field 9 length = 2

small position with energy type = 0x28
small position with energy field count = 11
small position with energy field 0 name = direction
small position with energy field 0 type = int
small position with energy field 0 length = 1
small position with energy field 1 name = timestamp
small position with energy field 1 type = int
small position with energy field 1 length = 2
small position with energy field 2 name = xpos
small position with energy field 2 type = int
small position with energy field 2 length = 2
small position with energy field 3 name = c2sping
small position with energy field 3 type = int
small position with energy field 3 length = 1
small position with energy field 4 name = bounty
small position with energy field 4 type = int
small position with energy field 4 length = 1
small position with energy field 5 name = pid
small position with energy field 5 type = int
small position with energy field 5 length = 1
small position with energy field 6 name = togglables
small position with energy field 6 type = int
small position with energy field 6 length = 1
small position with energy field 7 name = yvel
small position with energy field 7 type = int
small position with energy field 7 length = 2
small position with energy field 8 name = ypos
small position with energy field 8 type = int
small position with energy field 8 length = 2
small position with energy field 9 name = xvel
small position with energy field 9 type = int
small position with energy field 9 length = 2
small position with energy field 10 name = energy
small position with energy field 10 type = int
small position with energy field 10 length = 2

[Cerium]

Love the idea, hate the format.

I'm tired at the moment, but I'll post details tomorrow or something.

[Sharvil Nanavati]

I agree that standardizing on a packet list format is extremely
important. However, I am *strongly* in favour of XML over any other
format. It is the perfect format for describing the information that
we are working with (structured, hierarchical data), has excellent
validation tools, is human-readable, has ample manipulation tools
(XSLT for displaying the list is just one simple example), and has
fantastic language support so you don't have to write your own
(potentially buggy/incompatible) parser.

Bak, I'm wondering how you differentiate between control packets and
game packets using the format you describe above. They're two separate
protocols but they seem to be loaded from the same file.

I can't stress this enough: XML is the way to go here.

> ...
>
> read more »

[stanl...@gmail.com]

there's an "is core" value which I set to 1 on core (control) packets:

"cancel stream request iscore = 1"

also, packets that start with "incoming" are matched against incoming
packets if an outgoing version with the same type is defined. The only
thing I don't like about XML is that I have to use someone else's
library to parse it, whereas if it's plain text I can do it myself in
less time than finding the right library and figuring out how to use
it.

-Bak

On Jul 8, 8:13 pm, Sharvil Nanavati <Sharvil.Nanav...@gmail.com>
wrote:

> ...
>
> read more »

[Sharvil Nanavati]

Just as a suggestion for your project: it might be worthwhile to load
different protocols from different files instead of using flags like
'iscore'. Otherwise you'll end up with a plethora of flags and the
potential for easy semantic breaks. In the client, you'll want to
describe at least the control protocol, game protocol, directory
lookup protocol, ping protocol, and potentially other custom
protocols.

Using a custom format+parser is often a great solution for in-house
projects. You can develop at your own pace, you specify only what your
application needs, you can break compatibility whenever you please,
you can use something as simple as cin/fgets/fscanf, you get code
consistency because you're not depending on a 3rd party library that
defines its own data types, and so much more. I think it would be more
difficult to defend the use of XML if you wanted to use it just for
Discretion.

On the other hand, XML is the perfect choice for open exchange of
structured data. 3rd parties are no longer dependent on you to provide
accurate, reliable, timely, and authoritative descriptions of the file
format - the schema and the XML specification are the documentation.
There are XML libraries in just about every major language these days
- including JavaScript. What's more, if you're *really* against using
XML for your project, that's perfectly fine - you can write an XSLT
document to convert the input XML to the format your application uses
without ever having to code against a single XML library and again you
get all the benefits I mentioned above. The existing body of code, the
suitability of XML to this sort of data, the well-defined
specifications, and rich processing tools make XML The Perfect Choice
for this undertaking in my eyes.

Some general notes on how we, as a group, should specify any file
format to promote compatibility between applications:

If we use XML to represent some data, this group should take on the
responsibility of specifying the XML schema that the XML documents
must conform to.

If we define any proprietary file formats, this group should take on
the responsibility of specifying the language grammar, document
constraints (e.g. "a line cannot be more than 4096 bytes and the
parser must generate an error if a longer line is found in the
input"), and a reference implementation, in the public domain, of a
parser that accepts or rejects all input documents.

On Jul 9, 4:17 pm, "stanley...@gmail.com" <stanley...@gmail.com>
wrote:

> ...
>
> read more »

[Chris "Cerium" Rog]

I agree with the XML thing. Not only can it be transformed
(apparently), but it's very open and very readable to humans, plus can
support much more detail (in much less data) than most proprietary
formats.

[Sharvil Nanavati]

Okay, can we agree to XML? If so, let's come up with a reasonable
schema.