Something fundamentally wrong how localserver organizes it's files? (path length exceeded)
Tino
looking into the localserver directory in my FF profile via CygWin, I
noticed that I geht "filename too long" errors. As well I noticed,
that Gears became "somewhat slow" in loading recently added files to
the localserver.
The problem seems to be how localserver stores the files.
First, it stores about 500 files into one directory. That is OK.
So if it has to store more than 500 files, it creates a subdirectory
and puts the excess files there. That's OK, too.
However where it starts to become badly implemented is, when more than
1000 files are stored, as it creates the next subdirectory in the
subdirectory it created before. And so on. So with 50000 files you
have 100 subdirectories, each within the other, this is a path with
100 \ in it!
Not to say, that Windows must work through all this deep path each
time it tries to access a file, which makes it extremely slow, and
that all newer files are deeply buried in this extra long path.
What I would expect is, that the Localserver would organize files
differently, like, say, a logarithmic way. This is:
- Put the first 500 files into a subdirectory.
- Put the next 500 files into another subdirectory which may contain
500 other subdirectories. (this makes 25500 files)
- Put the excess files into a subdirectory which has 2 levels of
subdirectories. (this makes 525025500 files)
- Put the excess files into a subdirectory which has 3 levels of
subdirectories. (this should hold until 2050)
If you think this is not reasonable, then please think again.
Harddrives grow with factor 30 each decade. At the same time they
have shrunken in size and price dramatically. My first 1 GB drive in
1990 was about 1000 EUR and had 5.25" double height. Today I can buy
1 TB drives in 3.5" for less than 100 EUR.
In 2050 we will have 1000000 TiB drives - or whatever replaced them -
in our laptops - or whatever replaced them -. So there is nothing
wrong to have more than 1 billion files in a cache - for an ordinary
user.
Note that I would like to be able to cache 10 Million URLs in a Gears
localserver. TODAY (on my laptop).
But with the current implementation this just is not possible, as this
starts to become really uncomfortable around 100000 URLs. This is not
even enough for Wikipedia.
I will now try to create a CygWin based cleanup script to reorganize
the file structure to some sane layout. Then I will run this
periodically after adding files. I am not completely sure how this
works out, as I am not sure how Localserver creates the files then,
but we will see.
-Tino
Tino
The script mentioned now runs on my side under CygWin successfully.
However it probably is of no help for others, except for looking how I
did it, as I had no time to create a python script, so it was quickly
put together with bash and awk plus some of my own shell-tools. If
somebody is interested anyway please send an eMail, the script itself
is CLL, so not copyrighted, and therefor I will distribute it freely.
It works as follows:
It takes the files written by Gears localserver out of the old
location and moves them into a new location numbered MMM/KKK/FILE
where MMM is the ID/1000000 and KKK is the ID/1000%1000. It also
updates the database records accordingly. Now after some thousand of
files are added I run the script to cleanup the localserver directory
but keep the files accessible offline by gears.
The script is not perfect, as it sometimes leaves behind some files if
run concurrently to FF (else DB transaction would slow down everything
extremely), also it does not clean up abandoned directories yet.
The only problem left seems to be that FF slowstarts the localserver
as Gears seems to run through all database entries once at startup.
However afterwards everything is back to normal without delays imposed
by incomprehensibly long paths.
Currently I have over 90K files cached in a single localserver and the
DB uses around 1K per record. The script itself is prepared to work
up to 1G files, however I think before reaching this the SQLite DB of
Gears will slow down the FF startup beyond the time left for this
universe. However I expect that 10M files can be done on my laptop
now with some additional tricks.
Note that I do not think this hack is compatible to localserver
function removeStore, and resource functions remove, rename, copy. (I
did not check that ever.)
-Tino