Access specific objects from a binary file
Aaron J. M.
languages
but C++ is what I happen to be using for my latest project. First,
you
should know that I've never actually done any real serialization work
before. But anyway, I've been thinking about how my data/save files
would work. I have a vague idea about how one could pull out objects
by
reading a file (since I'd guess it'd be like reading values from a
text
file) but I don't know how I'd get a particular object from a file.
Say I only keep one map/level in memory. When the player leaves the
map, I have to pull the next map into memory from the save file. In
addition, I have to save the last level in the file as well. What I
can't get my head around is how, exactly, I'd know where the desired
map
is in the file.
Thanks for any help (again),
Aaron M.
Krice
> What I can't get my head around is how, exactly, I'd know where
> the desired map is in the file.
Save level files as temporary files during the gameplay, it's
easier to handle them.
Edwin DeNicholas
To keep it simple, you could save each level to it's own file.
Otherwise, if you really want one monolithic file, you might keep a
header at the beginning of the file which stores offsets (in bytes)
into the file where various interesting things can be found, such as
levels. While the game is running, you would just keep the header
information in memory.
Pfhoenix
no less than three major revisions of the serialization subsystem.
Adeo stores each map in its own file. Each map object, along with any
object whose class derives from Object, has a unique identifier called
UniqueObjectName (UON). The map's UON identifies the file it loads
from and saves to (the primary purpose of the UON is for named
identification for matching references between objects when
unserializing).
Adeo started with a binary save file format. I got tired of recreating
test maps every time something changed (binary compatibility is hard
to maintain in early development), so I then changed to an XML-like
format, but kept the basic serialization organization within objects.
My latest major change is restructuring serialization entirely to
support centralized, generic property processing (with the benefit of
enabling meta-data handling).
I advise strongly against a flat-file approach (mashing all the
serialized data and maps into a single file). If you want only one map
in play at a time, I see no reason to not go with a map-per-file
approach. Give the maps a unique identifier or name that coincides
with file name, and you've got an automatic name->file mapping method
right there.
- Martin
Gerry Quinn
@k37g2000hsf.googlegroups.com>, ajm...@ns.sympatico.ca says...
The simplest and almost certainly best way is to have a separate file
for each map/level - Crawl does this, for example.
Alternatively you could load and save the whole game, unless it's
gigantic.
- Gerry Quinn
--
Lair of the Demon Ape (a coffee-break roguelike)
<http://indigo.ie/~gerryq/lair/lair.htm>
Krice
> The simplest and almost certainly best way is to have a separate file
> for each map/level
I think it's clear by now. You can all stop repeating that.
Timofei Shatrov
tried to confuse everyone with this message:
>This is a C++ question. Well, I guess it could apply to other
>languages
>but C++ is what I happen to be using for my latest project.
...
>What I
>can't get my head around is how, exactly, I'd know where the desired
>map
>is in the file.
Use a serialisation library for your programming language. This is too
low-level problem for a roguelike developer to waste time on it.
--
|Don't believe this - you're not worthless ,gr---------.ru
|It's us against millions and we can't take them all... | ue il |
|But we can take them on! | @ma |
| (A Wilhelm Scream - The Rip) |______________|
Gerry Quinn
says...
> On Fri, 25 Apr 2008 10:43:37 -0700 (PDT), "Aaron J. M." <ajm...@ns.sympatico.ca>
> tried to confuse everyone with this message:
>
> >This is a C++ question. Well, I guess it could apply to other
> >languages
> >but C++ is what I happen to be using for my latest project.
> ...
> >What I
> >can't get my head around is how, exactly, I'd know where the desired
> >map
> >is in the file.
>
> Use a serialisation library for your programming language. This is too
> low-level problem for a roguelike developer to waste time on it.
What serialisation library do you recommend for C++, and what are the
benefits? I've always just implemented serialisation on a class by
class basis (except small POD-type classes are sometimes serialised by
the class that uses them) and it works fine and is not particularly
difficult.
Obnoxious User
> In article <4812f655....@news.motzarella.org>, gr...@mail.ru
> says...
>> On Fri, 25 Apr 2008 10:43:37 -0700 (PDT), "Aaron J. M."
>> <ajm...@ns.sympatico.ca> tried to confuse everyone with this message:
>>
>> >This is a C++ question. Well, I guess it could apply to other
>> >languages
>> >but C++ is what I happen to be using for my latest project.
>> ...
>> >What I
>> >can't get my head around is how, exactly, I'd know where the desired
>> >map
>> >is in the file.
>>
>> Use a serialisation library for your programming language. This is too
>> low-level problem for a roguelike developer to waste time on it.
>
> What serialisation library do you recommend for C++, and what are the
> benefits? I've always just implemented serialisation on a class by
> class basis (except small POD-type classes are sometimes serialised by
> the class that uses them) and it works fine and is not particularly
> difficult.
>
http://www.boost.org/doc/libs/1_35_0/libs/serialization/doc/index.html
--
OU
coppro
> http://www.boost.org/doc/libs/1_35_0/libs/serialization/doc/index.html
>
> --
> OU
Hey! You stole my post!
Also, the rest of Boost is awesome. Learn to use it well. Especially
shared_ptr, which will save you hours in memory management.
coppro
Gerry Quinn
@w5g2000prd.googlegroups.com>, rid...@gmail.com says...
I never felt the need for fancy pointers or serialisation libraries. I
generally don't pass pointers around, so I have no garbage collection
issues.
Jeff Lait
I don't know why everyone is such a big fan of saving individual maps
to disk. That made a lot of sense in 1980.
Keep all of your maps in memory at once. Save them all to disk every
time. When you actually get enough real content to make this
infeasible, then, maybe, consider adjusting your strategy.
If you want "save points" in case of program crash, explicitly save
all of your maps to disk on level change (or better yet, after X
moves) This, incidentally, makes creating a recovery program a lot
easier than trying to patch together individual map files.
With SaveScummer, I keep every map - and every previous version of
each map back to the beginning of the game - in memory without taxing
any realistic machine.
POWDER keeps all maps in memory and runs (item generation depending)
in a 256kb address space.
What's next? A discussion of how to use overlays to reduce code
footprint?
--
Jeff Lait
(POWDER: http://www.zincland.com/powder)
Pfhoenix
> in a 256kb address space.
How many maps does an average game of POWDER use/save?
- Pfhoenix
http://adeo.pfhoenix.com
Jeff Lait
> > POWDER keeps all maps in memory and runs (item generation depending)
> > in a 256kb address space.
>
> How many maps does an average game of POWDER use/save?
Like any sensible roguelike, it only creates maps on demand, so
"average" depends on your "average" player :>
The real question is how many maps does the maximal game of POWDER use/
save. This is 27. Each map is 32x32. Map storage requires two bytes
per tile (one for tile, one for flags) so is 2k per map. Map storage
thus eats 54k of the 256k available.
Nethack, for comparison, has I think around the order of 50 levels
with each one being 80x25, so around 100,000 tiles. This would put
the map overhead at about 200kb, explaining the lack of nethack for
the GBA :>
Taking an Asus EEE as a reference system, even doubling Nethack's map
memory, you are looking at one tenth of one percent of system RAM
being used for your map data.
Saving maps to disk during play just seems like such an unnecessary
complication. You introduce tons of painful edge conditions: what if
you run out of hard disk space while saving? What if someone plays
from a read-only hard drive mount? Or has their home directory marked
read only? Or deletes one of the files while playing?
Jürgen Lerch
On Tue, 29 Apr 2008 22:52:06 -0700 (PDT), Jeff Lait wrote:
> Saving maps to disk during play just seems like such an unnecessary
> complication. You introduce tons of painful edge conditions: what if
> you run out of hard disk space while saving? What if someone plays
> from a read-only hard drive mount? Or has their home directory marked
> read only? Or deletes one of the files while playing?
What do you do if you run out of RAM? (HD space is nowadays
at least as plenty as RAM.)
How does such someone save on exit?
And if a file gets deleted, well, you just create it anew.
Ad Astra!
JuL
--
jyn...@gmx.de / Work like you don't need the money
Jürgen ,,JuL'' Lerch / Dance like no one was watching
/ Love like you've never been hurt
/ - ?
Jeff Lait
> Saluton!
>
> On Tue, 29 Apr 2008 22:52:06 -0700 (PDT), Jeff Lait wrote:
> > Saving maps to disk during play just seems like such an unnecessary
> > complication. You introduce tons of painful edge conditions: what if
> > you run out of hard disk space while saving? What if someone plays
> > from a read-only hard drive mount? Or has their home directory marked
> > read only? Or deletes one of the files while playing?
>
> What do you do if you run out of RAM? (HD space is nowadays
> at least as plenty as RAM.)
As mentioned, modern OSes have virtual memory so running out of RAM is
not an option. Admittedly, 32bit OSes tend to have annoying 1.6gb,
2gb, or 3gb limits. But, in all honesty, if you don't have hard upper
cap of 1gb of memory free you do have problems.
Diskspace is actually more troublesome than ram. For example, you may
be running from a USB stick, or other system. I also rather
frequently run out of diskspace, but that's due to my tendency to
create gigabytes-per-hour data sets. Roguelikes shouldn't be doing
that.
A failure to memory allocation will cause damage at random parts of
your program, likely entailing a crash. A failure due to disk space
during your save method is a constrained problem you can deal with. A
failure due to diskspace in your scattered map loading/saving is much
more troublesome. There is no "ignore and continue" option - you have
to successfully save the map or the game is corrupt. Failing to save
a game to a single file doesn't have to corrupt the active game, so
all you have to do is report the failure and let the user try again at
their leisure.
> How does such someone save on exit?
How do you save on exit when maps are stored to disk? The problem is
the same - on exit you want your *current* level saved as well. If
your desire is to have checkpointing so you can recover from a crash,
I'd much sooner advocate checkpointing with an actual saved game
rather than trying to rebuild from the map data. This is conceptually
simpler to create, has only one code path, and removes any need to
make some special "recover" program.
> And if a file gets deleted, well, you just create it anew.
Either you have it in ram, so recreating it straightforward, but,
likewise, there's no good reason to have it on disk. Or you recreate
it by regenerating it. In which case this becomes a trivial cheat to
destroy troublesome encounters.
Jürgen Lerch
On Wed, 30 Apr 2008 05:00:12 -0700 (PDT), Jeff Lait wrote:
> On Apr 30, 8:33 pm, "Jürgen Lerch" <jyn...@gmx.de> wrote:
> > On Tue, 29 Apr 2008 22:52:06 -0700 (PDT), Jeff Lait wrote:
> > > Saving maps to disk during play just seems like such an unnecessary
> > > complication. You introduce tons of painful edge conditions: what if
> > > you run out of hard disk space while saving? What if someone plays
> > > from a read-only hard drive mount? Or has their home directory marked
> > > read only? Or deletes one of the files while playing?
> > What do you do if you run out of RAM? (HD space is nowadays
> > at least as plenty as RAM.)
> As mentioned, modern OSes have virtual memory so running out of RAM is
> not an option. Admittedly, 32bit OSes tend to have annoying 1.6gb,
> 2gb, or 3gb limits. But, in all honesty, if you don't have hard upper
> cap of 1gb of memory free you do have problems.
Well, rogue-likes aren't only played on modern computers and
modern OSes. I'm writing this on a 64+2 MB Amiga 4000. And
virtual memory never really caught on on the 'Miggy (but then
programs there never seemed *that* bloated).
But your Gameboy and Nintendo version should have similar or
worse RAM restraints (of course here I'm only refering to your
statement above that RAM's always aplenty; I guess those two
don't come with unlimited disk space either).
> Diskspace is actually more troublesome than ram. For example, you may
> be running from a USB stick, or other system. I also rather
Hm ... I see, with your method, if worse comes to worse,
you can play your game with persistent levels (with non-
persistent levels there's really no need to save single
levels to disk), you just can't save. With mine you can
still play the game (and not save), but only with non-
persistent levels.
So, if the whole game fits into memory, your method is
better for persistent levels, if not, well, not.
> A failure to memory allocation will cause damage at random parts of
> your program, likely entailing a crash. A failure due to disk space
> during your save method is a constrained problem you can deal with. A
Yes. So missing RAM is worse than missing disk space.
> failure due to diskspace in your scattered map loading/saving is much
> more troublesome. There is no "ignore and continue" option - you have
> to successfully save the map or the game is corrupt. Failing to save
No, the game isn't corrupt. As maps in rogue-likes are
usually created randomly anyway, at worst gameplay changes
from persistent to non-persistent levels.
> > How does such someone save on exit?
> How do you save on exit when maps are stored to disk? The problem is
> the same - on exit you want your *current* level saved as well. If
Yes. So having no write acces is a problem with both methods.
> your desire is to have checkpointing so you can recover from a crash,
> I'd much sooner advocate checkpointing with an actual saved game
> rather than trying to rebuild from the map data. This is conceptually
> simpler to create, has only one code path, and removes any need to
> make some special "recover" program.
I disagree. If your monolithic save file is damaged, the game
is gone, if one level file is damaged, only one level is lost.
> > And if a file gets deleted, well, you just create it anew.
> Either you have it in ram, so recreating it straightforward, but,
> likewise, there's no good reason to have it on disk. Or you recreate
> it by regenerating it. In which case this becomes a trivial cheat to
> destroy troublesome encounters.
Yes. So what?
Ad Astra!
JuL
--
jyn...@gmx.de / Reality is a crutch for those who can't
Jürgen ,,JuL'' Lerch / cope with fantasy
Krice
> Well, rogue-likes aren't only played on modern computers and
> modern OSes.
I think Jeff is trying to tell that if they are then there
is no need to save memory resources.
Jeff Lait
> Saluton!
>
> On Wed, 30 Apr 2008 05:00:12 -0700 (PDT), Jeff Lait wrote:
> > As mentioned, modern OSes have virtual memory so running out of RAM is
> > not an option. Admittedly, 32bit OSes tend to have annoying 1.6gb,
> > 2gb, or 3gb limits. But, in all honesty, if you don't have hard upper
> > cap of 1gb of memory free you do have problems.
>
> Well, rogue-likes aren't only played on modern computers and
> modern OSes. I'm writing this on a 64+2 MB Amiga 4000. And
> virtual memory never really caught on on the 'Miggy (but then
> programs there never seemed *that* bloated).
My thesis that you can drop '80s style programming techniques only
applies, obviously, if you drop '80s style hardware. So this is
hardly a counter example.
> But your Gameboy and Nintendo version should have similar or
> worse RAM restraints (of course here I'm only refering to your
> statement above that RAM's always aplenty; I guess those two
> don't come with unlimited disk space either).
The NDS has effectively unlimited diskspace. You can get a 1gb
microSD card for $7, for example. However, with 4Mb ram, the NDS has
effectively unlimited RAM for my purposes as well. The GBA really has
no choice in the RAM / disk trade off, the closest to "disk" is SRAM
which is a paltry 64kb.
> > failure due to diskspace in your scattered map loading/saving is much
> > more troublesome. There is no "ignore and continue" option - you have
> > to successfully save the map or the game is corrupt. Failing to save
>
> No, the game isn't corrupt. As maps in rogue-likes are
> usually created randomly anyway, at worst gameplay changes
> from persistent to non-persistent levels.
Persistent and non-persistent levels make for a completely and utterly
different gameplay.
Gerry Quinn
jyn...@gmx.de says...
> I disagree. If your monolithic save file is damaged, the game
> is gone, if one level file is damaged, only one level is lost.
I have heard of a technology known as "multiple save files" which is
said to be able to deal with this difficult problem.
Gerry Quinn
e73f03...@b1g2000hsg.googlegroups.com>, pau...@mbnet.fi says...
He is also pointing out that saving the whole thing is perfectly
feasible even on the little giblet of a device to which Powder is
targeted!
Jürgen Lerch
On Fri, 2 May 2008 15:56:59 +0100, Gerry Quinn wrote:
> In article <110786863251371...@news.freenet.de>,
> jyn...@gmx.de says...
> > I disagree. If your monolithic save file is damaged, the game
> > is gone, if one level file is damaged, only one level is lost.
> I have heard of a technology known as "multiple save files" which is
> said to be able to deal with this difficult problem.
Shock! Horror!
Isn't this evil technology a gaping permadeath-security hole?
(Are there indeed rogue-likes that use multiple save files?)
Btw., if I'm not mistaken, Nethack saves individual levels.
Ad Astra!
JuL
--
Paul Donnelly
> In article <110786863251371...@news.freenet.de>,
> jyn...@gmx.de says...
>
>> I disagree. If your monolithic save file is damaged, the game
>> is gone, if one level file is damaged, only one level is lost.
>
> I have heard of a technology known as "multiple save files" which is
> said to be able to deal with this difficult problem.
Is it really our job as game developers to anticipate the user's hard
drive crashing?
Gerry Quinn
jyn...@gmx.de says...
> Saluton!
>
> On Fri, 2 May 2008 15:56:59 +0100, Gerry Quinn wrote:
> > In article <110786863251371...@news.freenet.de>,
> > jyn...@gmx.de says...
> > > I disagree. If your monolithic save file is damaged, the game
> > > is gone, if one level file is damaged, only one level is lost.
> > I have heard of a technology known as "multiple save files" which is
> > said to be able to deal with this difficult problem.
>
> Shock! Horror!
> Isn't this evil technology a gaping permadeath-security hole?
Never really thought about that, to be honest. But then again, any save
file can be copied for backup purposes in most roguelikes. (To prevent
it would require some type of persistent state saving for the program
itself, sufficiently complex or awkward that the possibility of copying
the whole thing is limited.)
> (Are there indeed rogue-likes that use multiple save files?)
My little game Lair does, but then again I treat perma-death as
completely optional in it.
Edwin DeNicholas
What suggestions do you have for accessing specific data within a
single file? In the OP's example, we are looking for a specific map
within a single save file.
Gerry Quinn
@k1g2000prb.googlegroups.com>, disrup...@hotmail.com says...
Well, the OP has fallen silent on the issue, so perhaps he agrees with
the general advice that there is really no point trying to do that!
If you must do that, there are various options:
(1) Just read the file sequentially until you find the data
(2) Have some sort of simple and fixed file structure, for example store
each level at position 1000 + 50000 * nLevel, then start reading at that
point.
(3) Have a directory at some fixed point saying at what position in the
file everything is currently stored.
Note that (3) devolves essentially to reinventing a filing system that
stores multiple files. Your OS already has such a filing system - so
use that instead!
Pfhoenix
how many files are in the archive and a listing of name and offset
pairs so you can know where to jump to in order to start reading the
proper data. Adeo's texture archive works this way, essentially.
- Pfhoenix
http://adeo.pfhoenix.com
Aaron J. M.
> Well, the OP has fallen silent on the issue, so perhaps he agrees with
> the general advice that there is really no point trying to do that!
That's mostly because I'm not actually at the stage yet where I really
have to worry about serialization yet. I was just asking for the
purposes of planning. I figured the only-one-map-in-memory thing
would
be a usefull optimization, and I've noticed that a whole lot of games
seem to be able to keep their data in one save file.
Anyway, thanks for the help people.
Jeff Lait
Go straight to PhysicsFS: http://icculus.org/physfs/
This allows you to create a file system that lives in a single file.
David Damerell
>On Fri, 2 May 2008 15:56:59 +0100, Gerry Quinn wrote:
>>I have heard of a technology known as "multiple save files" which is
>>said to be able to deal with this difficult problem.
Trying to stop people save scumming is a tremendous waste of time; I like
POWDER's "Save Scum" option prominently displayed on the main menu. Make
it plain it's not how you're expected to play - but don't try and stop it.
One reason why not - relevant to this discussion - is that most of what a
player will do to deal with unreliable hardware is effectively
savescumming, and if you put technical barriers in the way you'll further
frustrate those poor souls.
--
David Damerell <dame...@chiark.greenend.org.uk> Kill the tomato!
Today is First Saturday, May - a weekend.