TextDB Performance

[Rebles]

Hey guys,

Another issue that I have been turning over and over in my head (when
I day dream, b/c I'm such a nerd), is TextDB Performance. As it
stands now, TextDB only grabs information from DB files when it's
needed, and caches the bare minimum. This method would be the best
type of database, if TDB was stored on a flash drive or a SSD, but
it's not. For traditional hard drives, it is inefficient to read
bytes of information in different parts of the file. TDB has many
examples: Searching one record at a time for queries (Each record
usually being a couple hundred bytes long); when reading the MEMO file
with a small block size, if it has to jump to different nonconsecutive
blocks in the file, this is inefficient. Additionally, the need to
call clearstatcache() in between TDB methods that alter the DB
structure (addField, editField, removeField), or else you lose data is
a big red flag. I would argue it would be more efficient to save the
changes in cache, and make the user call cleanUp(), which would push
the changes to the file.

I will list some pros and cons of caching changes. For starters,
performance would increase dramatically for scripts that performed
multiple queries on a table (A great example is the members table for
viewforum.php or viewtopic.php. Each user has to be queried for
simple things such as status, level, etc multiple times, and I notice
a dip in the script's performance b/c of it). When heavily edited a
table or records, performance will improve because the file is being
written to once. So, there will be significantly less reads and less
writes performing the same operations, which should improve
performance overall.

The flip side to this, is if a script calls a simple query, such as
get() or getNumberOfRecords(), which doesn't require reading in large
portions of the table. As long as we're smart about cache, I don't
see simple operations' performance drop. The key to performance is
knowing what and when to cache, and what and when not to cache.

TextDB has come a long way since the UPB one series. It has come a
long way since Tim first introduced it to the UPB Team to use for UPB
2.0 all those years ago. But like all products, in order to stay
competitive, you must continue to improve upon it.

--Rebles