Serious incompatibility with "Speedup My PC"
Martin Phillips
which will delete all QM files from the system.
We previously encountered the same problem with the EasyCleaner tool
in February 2007 and posted a warning notice.
The underlying issue here is that the algorithm used by the cleanup
tool is incorrect. Microsoft clearly document that a temporary file is
recognised by having the temporary attribute set and may
conventionally also have a named commencing with a tilde. It seems
that there are at least two cleanup tools that simply assume anything
starting with a tilde can be deleted.
We took this up with the authors of EasyCleaner in 2007 and they
accepted that they had disregarded the Microsoft specification but
said that they would not do anything about it because in their opinion
everyone knows that the leading tilde is all you need to check.
Our fix will be to rename the subfiles with a leading percent
character in place of the tilde. There will be a transition period
during which both are recognised by QM. They will be automatically
renamed on opening so long as the user has write access. There is one
small implication of this - the next incremental backup may be larger
than normal as all the renamed items will be included, not just those
that have been updated.
We hope to have a fix for this included in the next release of QM.
Until then, users should avoid the two cleanup utilities named above
and be aware that others may make the same wrong assumptions.
Ladybridge Systems Support
Ashley Chapman
2008/7/14 Martin Phillips <MartinP...@ladybridge.com>:
>
> We have been alerted to a problem with the "Speedup My PC" utility
> which will delete all QM files from the system.
>
> We previously encountered the same problem with the EasyCleaner tool
> in February 2007 and posted a warning notice.
>
> The underlying issue here is that the algorithm used by the cleanup
> tool is incorrect. Microsoft clearly document that a temporary file is
> recognised by having the temporary attribute set and may
> conventionally also have a named commencing with a tilde. It seems
> that there are at least two cleanup tools that simply assume anything
> starting with a tilde can be deleted.
>
What a load of garbage. An operating system that leaves temporary
files lying around, and disregard for the specification by the vendor.
And also by third party tools.
> We took this up with the authors of EasyCleaner in 2007 and they
> accepted that they had disregarded the Microsoft specification but
> said that they would not do anything about it because in their opinion
> everyone knows that the leading tilde is all you need to check.
More stupidity. This has really made my day!
>
> Our fix will be to rename the subfiles with a leading percent
> character in place of the tilde.
Why do you need a leading character at all? Why not just name the
subfiles 0, 1, 2? Or you could even use meaningful names like "hash",
"overflow" and "index"
Is there any compelling reason not to make this human (english)
readable? I ask, in part, because there is a very real risk that
percent character will foul up something else, such as backup scripts,
ftp etc.. I've recently had to rewrite part of an internal Anji
subroutine to remove ALL punctuation characters from a generated UUID,
because some web applications reacted badly to the percent character.
It's used as an escape character to include non-alpha-numerics in
URLs.
Please consider avoiding *ALL* punctuation characters in the subfile names.
> There will be a transition period
> during which both are recognised by QM. They will be automatically
> renamed on opening so long as the user has write access. There is one
> small implication of this - the next incremental backup may be larger
> than normal as all the renamed items will be included, not just those
> that have been updated.
>
> We hope to have a fix for this included in the next release of QM.
> Until then, users should avoid the two cleanup utilities named above
> and be aware that others may make the same wrong assumptions.
Do Windows system admins really run utilities like these without
either testing or taking a backup first?
>
>
> Ladybridge Systems Support
> >
>
--
Ashley Chapman
Martin Phillips
> Why do you need a leading character at all? Why not just name
> the subfiles 0, 1, 2? Or you could even use meaningful names
> like "hash", "overflow" and "index"
We have to distinguish between a directory that represents a dynamic hashed
file and one that represents a directory file. Each multivalue product that
has this problem has come up with a different solution. Unidata has a
different VOC entry for the two but this prevents implementation of
OPENPATH. UniVerse uses a hidden .Type30 item in directories representing
dynamic hased files. PI/open did something very similar to QM but used a
backspace as the leading character. Wow, did that cause troubles!
There is nothing to stop us using more meaningful names except for the need
to differentiate files as above. We could do this with something like %%data
and %%overflow instead of %0 and %1 but perhaps preserving the mystique
prevents users tinkering at the low level and causing troubles. The cryptic
naming tends to imply that these are internal items and should never be
touched directly by a user.
The change to use the new prefix, complete with automatic conversion, is up
and running here. We have much testing to do before we release this as it
will not be easy to revert.
Martin Phillips
Ladybridge Systems Ltd
17b Coldstream Lane, Hardingstone, Northampton, NN4 6DB
+44-(0)1604-709200
Kevin Powick
> Why do you need a leading character at all? Why not just name the
> subfiles 0, 1, 2? Or you could even use meaningful names like "hash",
> "overflow" and "index"
>
I read your reply to what Ashley posted, but I agree with him. It
would be best to avoid "funky" characters.
Why not just stick with standard naming conventions and put a suffix
on the file like 0.dat and 1.ovf?
I think curious users will examine files regardless of their naming
convention, so a cryptic convention is really not necessary. Also, I
too believe that using "odd" characters will lead potentially lead to
other problems.
--
Kevin Powick
Martin Phillips
> Why not just stick with standard naming conventions and put a suffix
> on the file like 0.dat and 1.ovf?
I have seen a UniVerse customer destroy his data through confusing the
system as to whether a directory was a hashed file or not. This cost him a
very large sum of money to recover as (amazingly) he didn't seem to have any
usable backups either.
The new system is up and running. The are no restricted uses of % as a
prefix and we have done this since 1993 in encoding forbidden characters in
directory file names so we don't expect problems.
Kevin Powick
wrote:
Hi Martin,
> I have seen a UniVerse customer destroy his data through confusing the
> system as to whether a directory was a hashed file or not. This cost him a
> very large sum of money to recover as (amazingly) he didn't seem to have any
> usable backups either.
confusing file names.
> The are no restricted uses of % as a
> prefix and we have done this since 1993 in encoding forbidden characters in
> directory file names so we don't expect problems.
software product decided to use it unconventionally? :-)
A quick search with Google found these potential problems using a %
char as part of a file name.
% Character has special meaning in URLs/URIs
MS Visual Source Safe - Not allowed in file names
MS SharePoint - Not allowed in file names
Cisco ECDN product - Not allowed in file names
Path filters on various FTP sites will not allow % char
Perhaps not relevant, but interesting, is that Nucleus does not
support % in dict files names
Since I was able to quickly find the above situations where % in the
filename is not accepted, I'm sure there are many more. I don't think
I really believe that exotic characters will protect a system from
stupid people, but I do believe that sticking with a well established
standard will minimize problems.
You could always go with something like:
QM.DATA.DO.NOT.TOUCH
QM.OVERFLOW.DO.NOT.TOUCH
Regards,
--
Kevin Powick
Martin Phillips
Thanks for your interest in this topic.
None of the restrictions that you list is relevant to this use of the
percent in filenames with perhaps the exception of FTP filters. Since these
are, from my reading of a web description, user configurable and not part of
the FTP specification, they should not affect us.
Both Windows and Linux document that the percent sign is valid in file
names. Of course, Windows also documents that the tilde is valid. The whole
cause of this issue is poor implementation of the two cleanup tools that
have chosen to treat the tilde as indicating a temporary file.
Whatever we do, the file name must be recognisably distinct from an item
that might be stored in a directory file. Hence the use of a prefix. Because
we already do name translation for forbidden charcters in directory file
records and we chose to use percent as the prefix for such translations, we
are completely in control of this. Any other system carries the risk of
clashing with names invented by a user.
Ultimately, whatever we do about this issue is up to us. Although it is
pleasing to have user input on this sort of change, the internals of the
file system are for us to manage and should be transparent to the user.
Ashley Chapman
2008/7/14 Martin Phillips <martinp...@ladybridge.com>:
It's clear you've thought this through. Can I make a suggestion though?
One of the main points to directory files is that it allows other
programmes to read and write to the directory. Although it's
unlikely, it is just possible that somebody could write the files "%1"
and "%2" to a directory; thus causing QM to think that the file has
become a dynamic file. You are not in control of this.
In order to adequately prepare the developer, I suggest that it's
clearly mentioned in the manual. Perhaps at this page:-
http://www.openqm.org/docs/directoryfiles.htm
That should keep everybody happy!
>
>
> Martin Phillips
> Ladybridge Systems Ltd
> 17b Coldstream Lane, Hardingstone, Northampton, NN4 6DB
> +44-(0)1604-709200
>
>
> >
>
--
Ashley Chapman
Martin Phillips
> One of the main points to directory files is that it allows other
> programmes to read and write to the directory. Although it's
> unlikely, it is just possible that somebody could write the files "%1"
> and "%2" to a directory; thus causing QM to think that the file has
> become a dynamic file. You are not in control of this.
Oh yes we are!! (well, reasonably so).
Just like UV, QM performs mapping of names that contain invalid characters
in directory files. To avoid the horrible incompatibilities that exist
between Windows and Linux in UV, we chose (rightly or wrongly) to use the
same translations on both platforms. Windows has the bigger set of
restrictions so we used this as our base. It is already documented at the
page that you referenced. Note how a percent sign gets translated to %P.
Therefore writing a record named %0 into a directory file yields an
operating system item named %P0 which is completely distinguishable from our
use of %0 in hashed files.
Just as in UV, but without the additional problem of type 1 file multi-level
name mapping, QM users who want to access items from outside QM either need
to program for these translations or, easier, simply avoid the restricted
characters.
So, why did I say "reasonably so"? We recently added the OSREAD and OSWRITE
operations to QMBasic. These allow a program to read or write an operating
system item with no name translation. Used with %0 as a name could cause
confusion but is somewhat unlikely. The whole point of these operations is
that they are totally transparent so the potential confusion is deliberate
though probably not an issue.
Kevin Powick
wrote:
> None of the restrictions that you list is relevant to this use of the
> percent in filenames
point is there are likely others.
> with perhaps the exception of FTP filters. Since these
> are, from my reading of a web description, user configurable and not part of
> the FTP specification, they should not affect us.
dealing. Often one does not.
> Whatever we do, the file name must be recognisably distinct from an item
> that might be stored in a directory file. Hence the use of a prefix.
unique to QM, so the problems you hope to avoid could still happen.
> Ultimately, whatever we do about this issue is up to us.
at this time does not pose any problems for me. I'm just pointing out
that your solution is likely not unique, nor without its potential
issues -- As may be the case with almost any naming convention.
--
Kevin Powick
Martin Phillips
I would really like this to end with us being in agreement.
The only file system items affected by this change are the subfiles that
make up a dynamic hashed file. The percent sign is documented for both
Windows and Linus as being valid in names so any restrictions are imposed by
particular uses such as our own, not by the file system itself.
If a user is sending a dynamic hashed file via FTP it seems a fairly valid
assumption that he is in control of what happens at the other end as it is
otherwise meaningless junk.
I really don't think we have a problem. We have used the percent sign for
special purposes in QM since 1993 without problems. All we are doing here is
adding another use.
Ashley Chapman
2008/7/14 Martin Phillips <martinp...@ladybridge.com>:
>
> Hi Ashley,
>
>> One of the main points to directory files is that it allows other
>> programmes to read and write to the directory. Although it's
>> unlikely, it is just possible that somebody could write the files "%1"
>> and "%2" to a directory; thus causing QM to think that the file has
>> become a dynamic file. You are not in control of this.
>
> Oh yes we are!! (well, reasonably so).
Oh, no you aren't!! (in panto style, despite it being too early in the year)
This is obviously not as clear as I think it is. If I write a perl,
cobol, Pascal, C++, Java or other programme, I can write into the
directory file, and QM has no control of this. This is the point of
directory files, to allow for other systems to link and interact.
Even if it's only notepad to edit BASIC programmes.
Is it really worth arguing about it any more, when a simple addition
to the documentation will help prevent people from being caught out in
the future?
--
Ashley Chapman
Martin Phillips
> This is obviously not as clear as I think it is. If I write a perl,
> cobol, Pascal, C++, Java or other programme, I can write into the
> directory file, and QM has no control of this. This is the point of
> directory files, to allow for other systems to link and interact.
> Even if it's only notepad to edit BASIC programmes.
OK, I had been thinking in the reverse direction. Yes, you could mess up the
file system in this way though if you chose to write an item named %0 all
that would happen is we would report the directory as being a corrupt DH
file.
The whole think will be clearly documented so hopefulyl there will be no
problem. Certainly no one has tried to write an item named ~0 in the past.
End of thread?
Kevin Powick
wrote:
Hi Martin,
> I would really like this to end with us being in agreement.
> The percent sign is documented for both
> Windows and Linus as being valid in names so any restrictions are imposed by
> particular uses such as our own, not by the file system itself.
system. It's about other applications and their interpretation of
filenames containing anything beyond [A-Z], [a-z], [0-9], dot, dash,
and underscore. Hasn't the sole impetus for this change been this
very issue? -- The incorrect interpretation of the currently used
tilde by several different applications?
> If a user is sending a dynamic hashed file via FTP it seems a fairly valid
> assumption that he is in control of what happens at the other end
corporate systems. Often we are only dealing with a department in a
larger entity.
> I really don't think we have a problem. We have used the percent sign for
> special purposes in QM since 1993 without problems. All we are doing here is
> adding another use.
me. If you've never run into difficulties using percent signs in file
names since '93, then you have much more experience than I do, and I
trust your judgement. It's just something I would never do.
As a funny aside, note that people coming from several Pick flavors
are used to interpreting percent signs as tape parity errors. :-)
Does anyone use tape anymore?
Regards,
--
Kevin Powick
Steve Bush
them thinking they are overlay files containing binary. Configuring
exclusions is a pain or even impossible. Lessons from AREV.
Martin Phillips
End of thread. The job is done. It has passed all our tests.
The new subfile names will be, as initially proposed, %0, %1, etc.
This works. It doesn't lead to ambiguities between hashed files and
directory files. It will be in the next build.