Instead of inventing you're own wheel, you could perhaps extend an
OGUP is a simple engine that crawls the gophernet, only to keep a list
of active servers (and possibly discover new servers). In the process
it collects the content of directories, but throw it away. It would be
relatively easy to make it write the content of directories into some
database (relational or filesystem-based) for archiving purposes. Then,
extend the archiving activities to text files. I'd gladly provide you
with pointers if you'd like to work on that.
OGUP is written in C89.