PKWare files Patent - .zip is dead?
Darryl Lovato
released their own variant methods of adding strong encryption to the
zip standard, which is bad enough - neither is compatible with each
other, and both are obviously not backward compatible with the thousands
of zip implementations out there already.
It get's worse now, because PKWare has apparently filed a patent that
supposedly covers this. I guess it is an attempt to block WinZip's way
of doing it, and potentially to start charging fee's to those of us who
want to stay compatible with the .zip file format. Both companies
appear to be fighting to be the "owner" of the .zip file format, but
IMHO, the day that Phil Katz released the tech specs to the world, the
user community became the owner of the .zip format.
I haven't been able to see exactly what PKWare is claiming in the
patent, but there is certainly a lot of prior art with respect to
encryption + compression - StuffIt, for example has had this for a long
time...
Anyway, I don't know who will win this, but I do know who is going to
loose - users. About the only thing the zip file format had going for
it (since it's so extremely out of date in many areas) was the fact that
you knew that no matter who you sent a .zip file to, you could be
assured they could open it.
This is no longer the case.
Now WinZip .zip files can only be opened by winzip, and PKZip .zip files
can only be opened by PKZip, and if either of these are sent to any of
the thousands of other zip implementations, they cant be opened - and to
your average user - if they can't open the file - it is corrupt.
Both companies could have made this much better for users - by changing
the extension of incompatible .zip files to something different (.pkzip,
.winzip, .zip2, etc) so that your average user would recognize the
file's different.
My guess is you are now going to see a "Free for all" of companies
releasing .zip file implementations that are not compatible with each
other - some may add better compression, or other features. The
precident has been set by PKWare/WinZip - change the format all you
want, and keep the .zip extension the same. In other words, the ".zip"
file format is now irrevicably fractured, and it's sole benefit
(umbiquity) has been lost forever.
Here's the news article on the Patent application...
http://www.pcworld.com/news/article/0,aid,111758,00.asp
Anyone have any thoughts?
- Darryl Lovato
CTO, Aladdin Systems, Inc
Arild Bjørk
news:dlovato-991D82...@newssvr21-ext.news.prodigy.com...
......
>
> Here's the news article on the Patent application...
>
> http://www.pcworld.com/news/article/0,aid,111758,00.asp
>
> Anyone have any thoughts?
>
> - Darryl Lovato
> CTO, Aladdin Systems, Inc
By the way, who are using the PKWare products. I downloaded a test version
of the most basic version and my impression is that it's not interesting. It
supports very few formats and as a WinZip and a Powerarchiver user, there
are better alternatives. PKware products are simply not interesting for the
average user.
Carsten Hansen
"Darryl Lovato" <dlo...@aladdinsys.com> wrote in message
news:dlovato-991D82...@newssvr21-ext.news.prodigy.com...
.
.
> Anyway, I don't know who will win this, but I do know who is going to
> loose - users. About the only thing the zip file format had going for
> it (since it's so extremely out of date in many areas) was the fact that
> you knew that no matter who you sent a .zip file to, you could be
> assured they could open it.
>
> This is no longer the case.
>
> Now WinZip .zip files can only be opened by winzip, and PKZip .zip files
> can only be opened by PKZip, and if either of these are sent to any of
> the thousands of other zip implementations, they cant be opened - and to
> your average user - if they can't open the file - it is corrupt.
>
.
.
> - Darryl Lovato
> CTO, Aladdin Systems, Inc
Your claim is exaggerated. Only files that are encrypted are tied to the
program that created them.
Using encryption for archiving is of dubious value. Do you remember the
password five years later?
If you always use the same password, you run the risk of a global break.
For file transfer you have the problem of how to exchange the password
securely.
Personally I don't see this as taking off.
Carsten Hansen
Darryl Lovato
<2DfUa.72679$0v4.4...@bgtnsc04-news.ops.worldnet.att.net>,
"Carsten Hansen" <hans...@worldnet.att.net> wrote:
> "Darryl Lovato" <dlo...@aladdinsys.com> wrote in message
> news:dlovato-991D82...@newssvr21-ext.news.prodigy.com...
>
> .
> .
> > Anyway, I don't know who will win this, but I do know who is going to
> > loose - users. About the only thing the zip file format had going for
> > it (since it's so extremely out of date in many areas) was the fact that
> > you knew that no matter who you sent a .zip file to, you could be
> > assured they could open it.
> >
> > This is no longer the case.
> >
> > Now WinZip .zip files can only be opened by winzip, and PKZip .zip files
> > can only be opened by PKZip, and if either of these are sent to any of
> > the thousands of other zip implementations, they cant be opened - and to
> > your average user - if they can't open the file - it is corrupt.
> >
> .
> .
> > - Darryl Lovato
> > CTO, Aladdin Systems, Inc
>
>
> Your claim is exaggerated. Only files that are encrypted are tied to the
> program that created them.
People use encryption when sending files (e-mail, ftp etc) as well as
"backup/archiving", and they are using encryption more and more because
of security concerns. All of this is moot if the files don't leave the
users machine - the program that created them can undo them, but zip
files are exchanged as much if not more than they are used for backups.
i.e. ".Zip" files rarely stay put on one machine, which is what creates
the problem.
> Using encryption for archiving is of dubious value. Do you remember the
> password five years later?
And now you not only have to rember what the password is, but you have
to remember which product you used to create it.
> If you always use the same password, you run the risk of a global break.
> For file transfer you have the problem of how to exchange the password
> securely.
Yep. And now you also need to make sure the recipient has the same zip
program that you used to create the .zip file as well. If you have
PKZip, you can't send an encrypted archive to a WinZip user (and vice
versa). If you create a encrypted file with either, the built in zip
support in windows won't be able to decode the file - which for all
intents and purposes will appear to be "corrupted" by the average user.
> Personally I don't see this as taking off.
We'll see. IMHO, has the potential of becomming a real mess.
- Darryl
> Carsten Hansen
>
>
Darryl Lovato
"[ Doc Jeff ]" <m...@privacy.net> wrote:
> Yes. Normally I don't bother correcting people anymore because it has
> become boring. But you represent a company that I actually respect and
> I'd rather see you learn about your language errors now than risk your
> company's reputation.
Sorry. I was in a hurry to get the original message posted before I
went into a meeting this morning. My apologies.
>
> As to the content of your message, I agree. My personal feeling is
> that ZIP has become obsolete. I don't make ZIP files these days
> because other compressors do a better job of compressing files. I
> grant that ZIP is the most well-known archive format but others are
> surpassing it both in terms of quality as well as technology.
>
> It is time to stop shoehorning new tech into an old format and start
> innovating a new format.
Aladdin faced a similar issue with the StuffIt format about 5 years ago.
We had added-on, then added-on, and then added some more. Although the
user features became better, and we kept up with algorithm research, it
became too much to support and became clunky.
We decided to throw out the whole source base and start a full re-write.
It took a few years with a fairly large engineering team, but now we
have fully modern, extensible, code base, a new state of the art archive
format (.sitx - we changed the extension), and retained the ability to
handle all the older formats - zip, sit, etc, etc.
The "Zip companies", on the other hand, have done basically nothing
(Zip64, and the BZip algorithm addition are small exceptions - which are
still not universally supported) to improve Zip compression in the last
10+ years, and have focused their efforts on marketing and user
interface features - Ignoring the reason people use the product in the
first place - to make file's smaller. The Zip format has become
static/dead with no real innovation because nobody took ownership to
advance the format after PKWare put it into the public domain.
One problem with the .zip format, is that nobody owns it, and given the
recent war between PKWare and WinZip, I seriously doubt that they would
be able to come to a consensus on what a new format should be. Phil
Katz (the inventor of .zip) would have been the best person to lead
this, but he passed away a couple years ago. Even if the zip community
could come to consensus, it would take years before it would be rolled
out - unless a group has been secretly working on this for the last
couple years.
Further, since there are so many unique implementations of the zip
format, (hundreds of developers have rolled their own), it would be
almost impossible to have a coordinated roll out of a new zip format -
some programs would support it, others wouldn't, etc. It would be a
real mess for users.
Maybe Aladdin should have named ".sitx", ".zip2" :-)
- Darryl
Rainer Nausedat
ext.news.prodigy.com>, dlo...@aladdinsys.com says...
> The "Zip companies", on the other hand, have done basically nothing
> (Zip64, and the BZip algorithm addition are small exceptions - which are
> still not universally supported) to improve Zip compression in the last
> 10+ years, and have focused their efforts on marketing and user
> interface features - Ignoring the reason people use the product in the
> first place - to make file's smaller.
Pkzip (Deflate32) is doing a good job, especially if one take the
compression speed into account. Pkzip is lightening fast and beats
allmost all competitors. Beside this, it seems that there are a lot
of users who do not want *pay* compression improvements with plenty
of time. For daily backup tasks I would always prefer a fast
archiver.
> The Zip format has become static/dead with no real innovation
> because nobody took ownership to advance the format after PKWare
> put it into the public domain.
May be they were not much interested in such improvements for the
above reasons. Pkware were always the stewart of the zip format (at
least they call themselves the stewart of the zip format) and I
assume that they have enough money to develope or buy a better
compressor.
Also, the good folks from the Info-Zip group contributed a lot of
enhancements to the zip format.
> Maybe Aladdin should have named ".sitx", ".zip2" :-)
Dou you really think it would be different then? Please correct me
if I'm wrong, but I couldn't find any specs/white papers on the sit
archive format. I have tried a couple of times to contact Aladdin
because of that, and not even a response from Aladdin till now.
R.Nausedat
John Smith
"Darryl Lovato" <dlo...@aladdinsys.com> wrote in message
news:dlovato-991D82...@newssvr21-ext.news.prodigy.com...
Mikael Lundqvist
> Perhaps Aladdin should document the sitx format and open it up?
>
The sitx format is vastly superior to zip, it offers compression close
to the very best of today's compressors.
Sure, I'd be interested in a description of the algorithms which are
being used in it.
An educated guess (based on results and speed) would be PPM with filters
like delta-coding (it did very well on kennedy.xls, 13,686 bytes - 0.106
bpc, and lcet10.txt, 90,388 bytes - 1.694 bpc).
--
Mikael Lundqvist
http://www.geocities.com/mikaellq/
Occam's Razor:
"Keep things simple!"
Sachin Garg
I agree with you that not-many are using PKWare products, but
a-lot-many are using ".zip" and if anyone (like PKWare) gets a patent
on it, it effects them all.
Important issue here is that we will soon be having files with
*different* formats floating around with ".zip" extension.
Sachin Garg [India]
http://sachingarg.go.to
http://sachingarg.cjb.net
Darryl Lovato
Rainer Nausedat <RNau...@speedproject.de> wrote:
> In article <dlovato-10E03D.14425625072003@newssvr24-
> ext.news.prodigy.com>, dlo...@aladdinsys.com says...
>
> > The "Zip companies", on the other hand, have done basically nothing
> > (Zip64, and the BZip algorithm addition are small exceptions - which are
> > still not universally supported) to improve Zip compression in the last
> > 10+ years, and have focused their efforts on marketing and user
> > interface features - Ignoring the reason people use the product in the
> > first place - to make file's smaller.
>
> Pkzip (Deflate32) is doing a good job, especially if one take the
> compression speed into account. Pkzip is lightening fast and beats
> allmost all competitors.
Deflate is fast, but other algorithms can compress MUCH better. As
machines have increased speed, the complexity of the algorithms is less
of an issue.
On any "modern" machine compressing an average size 200K file, you won't
be able to see any difference between a "slow" algorithm (bwt/ppm, etc),
and a "fast" one (lz) - can you tell the difference between .001 second
and .01 second? I can't - but the ppm compressed file might be 20-30%
smaller. (p.s. I just made these numbers up as a reasonable example,
don't ask for benchmark files - lots of web sites exist for compression
testing...)
> Beside this, it seems that there are a lot
> of users who do not want *pay* compression improvements with plenty
> of time.
There are always users who want something or everything for free. We
give away our decompressor, but if you want to compress, it's either
shareware, or for the full feature set - commercial. We have to be able
to pay our engineering staff on payday.
> For daily backup tasks I would always prefer a fast
> archiver.
It depends on what you are compressing - you CAN see a difference in
speed if you are backing up a whole machine (gigabytes of data). But
even then, you are focusing on the compression algorithm - there is a
lot more to an archiver/format (.sit, .zip, etc) than just the
compression algorithms. In your backup case, wouldn't it be nice to
have error correction built into the archive - .sitx has error
correction, .zip doesn't. Lots of other issues come into play like
multi-platform support, and others. Even simple things like
blockmode/solid mode which the more modern archivers have, are not
possible in the zip format.
It's not that zip is bad, it's just old, and showing signs of that age.
I don't think it is reasonable to expect that the current zip format
would live forever. It's a great thing that it has lived as long as it
has - it's just nearing it's end of life.
> > The Zip format has become static/dead with no real innovation
> > because nobody took ownership to advance the format after PKWare
> > put it into the public domain.
>
> May be they were not much interested in such improvements for the
> above reasons.
Maybe. I have no idea why they didn't innovate with the format for 10+
years, all I know is they didn't. Perhaps the stuff that Phil Katz was
going through before his death had something to do with it (very sad),
but I really don't know for sure.
> Pkware were always the stewart of the zip format (at
> least they call themselves the stewart of the zip format) and I
> assume that they have enough money to develope or buy a better
> compressor.
PKware was the stewart of .Zip until Phil decided to release the format
to the public - they did a couple minor things after that time, but not
much. There isn't much incentive to put engineering money into
something you have to give to your competitors.
> Also, the good folks from the Info-Zip group contributed a lot of
> enhancements to the zip format.
Yes they did, but they seem to have stopped as well - nothing new from
them in a long time.
> > Maybe Aladdin should have named ".sitx", ".zip2" :-)
>
> Dou you really think it would be different then?
It was sort of a joke, but it might have made our format a much clearer
alternative to the current zip format, and maybe more people would
realize it would be a good replacement.
> Please correct me
> if I'm wrong, but I couldn't find any specs/white papers on the sit
> archive format. I have tried a couple of times to contact Aladdin
> because of that, and not even a response from Aladdin till now.
We don't publish the format details or source code - but we do have low
cost developer libraries, which a lot of companies use for not only .sit
support, but for .zip and a bunch of others.
There are a number of reasons we dont give out format specs (see what
happened to pkware and is happening to the .zip format right now for
some reasons why it may not always be a good idea).
We look at the benefits/drawbacks of releasing specs every couple years
- every time we've looked at it so far, it's the .sit format and it's
users would be better off if we kept control of the direction of the
format. We could have a whole thread on the pluses/minuses of open
source.
- Darryl
> R.Nausedat
Darryl Lovato
"John Smith" <som...@somewhere.com> wrote:
> Perhaps Aladdin should document the sitx format and open it up?
Perhaps, but every time we've looked at that option in the past (and we
do regularly) the upside is outweighed by the downside. If you don't
think there are any downsides to doing open source - just look at what
happened to PKWare and what's going on with ".zip" right now.
We make sure that we have reasonably priced developer libraries (so
developers can integrate support), we have free decompressors for the
common platforms, a shareware compressor, and users who want the full
feature set can buy the commercial version.
- Darryl
Rainer Nausedat
ext.news.prodigy.com>, dlo...@aladdinsys.com says...
> Deflate is fast, but other algorithms can compress MUCH better. As
> machines have increased speed, the complexity of the algorithms is less
> of an issue.
>
> On any "modern" machine compressing an average size 200K file, you won't
> be able to see any difference between a "slow" algorithm (bwt/ppm, etc),
> and a "fast" one (lz) - can you tell the difference between .001 second
> and .01 second? I can't - but the ppm compressed file might be 20-30%
> smaller. (p.s. I just made these numbers up as a reasonable example,
> don't ask for benchmark files - lots of web sites exist for compression
> testing...)
Well, may be my last statement was somewhat unclear as english is
not my native language.
No, I won't feel the difference between 0,01 second and 0,1 second.
But lots of our customers are dealing with backups (compressed) >
10 GB or > 250.000 files. Those will really feel the difference
between a "fast" and "slow" compressor/archiver.
> > Beside this, it seems that there are a lot
> > of users who do not want *pay* compression improvements with plenty
> > of time.
>
> There are always users who want something or everything for free. We
> give away our decompressor, but if you want to compress, it's either
> shareware, or for the full feature set - commercial. We have to be able
> to pay our engineering staff on payday.
Sorry, I wrote "users who do not want *pay* compression
improvements with plenty of time". It does not mean that these
users are looking for free software, it means only that the
archiving/compression time is also an issue.
> > For daily backup tasks I would always prefer a fast
> > archiver.
>
> It depends on what you are compressing - you CAN see a difference in
> speed if you are backing up a whole machine (gigabytes of data). But
> even then, you are focusing on the compression algorithm - there is a
> lot more to an archiver/format (.sit, .zip, etc) than just the
> compression algorithms. In your backup case, wouldn't it be nice to
> have error correction built into the archive - .sitx has error
> correction, .zip doesn't. Lots of other issues come into play like
> multi-platform support, and others. Even simple things like
> blockmode/solid mode which the more modern archivers have, are not
You are right here, especially built in error correction is a big
plus. I were begging Pkware to add such a feature, again, no
answer. So we added a compatible solution by ourselves as APPNOTE
leaves enough room for such a solution.
Don't get me wrong here, I don't say that the zip archive format is
the final solution. The zip archive format is really not up to
date, but it has still some advantages... (more at xxx)
> > Also, the good folks from the Info-Zip group contributed a lot
> > of enhancements to the zip format.
> Yes they did, but they seem to have stopped as well - nothing new from
> them in a long time.
Believe me, more to come...
> > Please correct me
> > if I'm wrong, but I couldn't find any specs/white papers on the sit
> > archive format. I have tried a couple of times to contact Aladdin
> > because of that, and not even a response from Aladdin till now.
> We don't publish the format details or source code - but we do have low
> cost developer libraries, which a lot of companies use for not only .sit
> support, but for .zip and a bunch of others.
> There are a number of reasons we dont give out format specs (see what
> happened to pkware and is happening to the .zip format right now for
> some reasons why it may not always be a good idea).
xxx:
I think it would be rather difficult to establish a complete
"closed" archive format. BTW, I'm not talking about open source
here, I'm talking about public, freely available specs/white papers
on an archive format (sit). Even a freely available decompressor in
source code would be sufficient since
- people may not trust pre-build libraries
- people may need for legal reasons (for example use of a software
in an army or gouvernement departement) the source code
> - but we do have low cost developer libraries...
So, what happens if we license your SDK to create/edit/extract sit
archives on Win32/MAC/Linux machines and our app would blow Stuffit
away from the market (even that I would not exspect this)? Would
Aladdin renew our license?
> ... but we do have low cost developer libraries, which a lot of
> companies use for not only .sit support, but for .zip and a bunch of
> others.
Now I'm really curious. Our own libraries support all common
archivers except Stuffit (sit). So I'll give your sales departement
another try.
R.Nausedat
John Smith
not, I should clarify that I wasn't implying you should open the source code
to stuffit, just releasing the file format - that is not open source.
"Darryl Lovato" <dlo...@aladdinsys.com> wrote in message
news:dlovato-6F694D...@newssvr21-ext.news.prodigy.com...
Darryl Lovato
> By 'open source', I'm assuming you are referring to 'open specification'. If
> not, I should clarify that I wasn't implying you should open the source code
> to stuffit, just releasing the file format - that is not open source.
If it's enough information that someone could create source from it,
then it might as well be source - PKWare didn't give out source, but
enough info that a reasonably savvy developer could create source from
the description - which WinZip did...
There are some pretty cool things we are doing in the file format
itself, some of which took a lot of money/engineering effort to create -
why should we give this away? We are not a charity - we invest money in
our engineering so that we have an edge on the competition - sometimes
the engineering work pays off, sometimes it doesn't - we do enough to
stay in business, and have some growth most years. If we're lucky each
year we have some profit - to put back into next years development -
occasionally we loose a money. What we've been doing has worked for 15
years - not many "tech" companies can say that. We've served our
customers pretty well, adapted to new technology, and I think done a
pretty good job of improving the interface, and product in general.
We haven't filed patents - even though we have a couple dozen ideas in
the product that could be patented. If someone else comes up with a
similar idea - on their own - then good for them, we are not going to
try to block other developers from innovating, but we are also not going
to hand them our hard work on a silver platter. If we gave out the
format details, source etc, we would be in the same boat as
PKware/WinZip - there would be no incentive for us to improve the
StuffIt format - it would just gather dust for 10 years like the .zip
format did. We'd be competing based on who has the best U/I or
whatever, and the core of the product would sit idle.
So the question is why would users benefit by us releasing the source -
what are they going to do with it? The only people that benefit from
public source are the developers who want to use the original developers
hard work to compete against them. Users can loose in the whole deal,
because they end up with:
* Multiple variations of what was once a single format (see current zip)
* No incentive for the creator of the format to continue to bring it
forward (see zip)
* No easy way to coordinate a rollout for any improvements (see zip)
* A bogged down change process that may require different companies,
each with different goals trying to define each revision. (see mp3,
jpeg, etc)
* Possible ruin/elimination of the visionary/creator of the format,
leaving nobody to move it forward (see zip).
There's more, but this should be enough to at least show that there is a
downside to publishing source - not just for the "company", but for the
long-term good of the "format" and it's "users".
- Darryl
Stuart Caie
> Anyone have any thoughts?
Yes, I do.
While the original ZIP format has limitations on things like overall file
size and number of files in the archive, new ZIP products shouldn't
automatically generate "enhanced" ZIP files unless they actually need to
exceed the old format's limitations. Deliberately trying to break
compatibility in order to get people to "upgrade" is a rotten tactic.
If I were to use strong encryption in archiving, I would archive my files
with an archiver such as RAR, StuffIt, PKZIP or 7-Zip, then I would store or
send the archive using a trusted and reliable encryption system like PGP,
rather than trust to any proprietary "in built" encryption of the archivers.
I might make an exception for 7-Zip and RAR. 7-Zip is free software, RAR
provides the entire UnRAR source code freely, but only licensed for
examination and UnRAR porting purposes, not for general reuse. With both
these archiver sources, I can examine both the crypto algorithms used and
the way they are implemented in the product, and be sure that what I see is
what I use by compiling the sources myself.
I've have always wondered why Aladdin Systems have never released any
specifications for their StuffIt file formats, or any of the compression
methods like Arsenic.
One of the greatest advances for compression in software engineering was the
release of zlib, based on PKWARE's generous donation of the deflate
algorithm to the public domain. deflate and inflate are now implemented in
thousands, possibly millions of hardware and software products. They are
even an Internet standard, providing transparent compression of WWW pages,
for example. Being an LZH method, deflate still strikes a good middle ground
between speed and compression ratio.
ZIP is not only very popular, but its file format specifications are well
documented. Any program can add basic ZIP reading and writing functionality,
even in the oddest places -- for example, simply dropping a zip file in
WinAmp's skin directory is enough to use a skin. WinAmp unpacks the zip file
itself.
Software writers have a choice of implementing ZIP functionality themselves,
with the help of zlib, or with the help of many other free, shareware and
commercial compression libraries. Also, ZIP and deflate will remain open and
documented even if your ZIP library vendor disappears.
I doubt many people will want to move away from the old ZIP format. Its
competitors and successors don't have anything like the same openness and
cross-compatibility.
Regards
Stuart
Mark Adler
> Both companies
> appear to be fighting to be the "owner" of the .zip file format, but
> IMHO, the day that Phil Katz released the tech specs to the world, the
> user community became the owner of the .zip format.
Actually, Phil Katz quite explicitly and intentionally made both the
".zip" extension and the zip format public domain. He also committed
to updating the PKZip application note, which describes the format, as
the PKZip product evolved. That promise was kept while he was alive.
Now however, PKWare appears to want to make parts of the format a
trade secret, which as you point out completely undermines what makes
the .zip format useful in the first place. In addition to the
encryption, they have also declined to document the deflate64 format
in their application note, despite at least two revisions of that note
since deflate64 was introduced. In this case, it turns out to be not
very difficult to reverse engineer the format. However the corporate
intent is clear. The corporate intent is also self-destructive.
So, now may be the time for the community, in particular the community
that reads this newsgroup, to develop an open, scalable cross-platform
format that supports archives of directory structures, files, and
meta-data, high-quality lossless compression, and high-quality
encryption and authentication. "Cross-platform" does not mean
"Windows and Mac", but rather as wide a range of platforms as there
are contributors. The PNG format effort is in my opinion a good model
for this sort of development. (I played a small part in that
development.)
A difficulty with this concept is that the development of high-quality
compression over a wide range of types of data requires a great deal
of time, determination, and expertise--perhaps more so than one should
expect to achieve in contribution to a free, open-source effort.
Therefore I might suggest a compensation scheme where corporate users
of the software would be obligated to contribute directly to the
authors of the compression/decompression methods that they use. This
would encourage the development of better compression methods over
time, in whatever dimensions are of interest to the paying users
(space, time, specialized models for specific data, etc.). How it
would be decided when to add a new method to the official format is
left as an exercise for the reader. Also whether or not to accept
methods with patented components, licensed for free use, is left for
the reader to ponder. In any case, as much thought would probably
have to be put into the business and legal model as is put into the
format itself.
I am posting this idea merely to stimulate discussion. I personally
don't have the time or inclination to play a major role in such a
development. (My day job is both interesting and time-consuming.)
But if a good group is motivated to do so, and can produce on a
schedule, I'm thinking on the order of 12 to 18 months, everyone will
benefit greatly in the long run.
Mark Adler
Mark Adler
> One of the greatest advances for compression in software engineering was the
> release of zlib, based on PKWARE's generous donation of the deflate
> algorithm to the public domain. deflate and inflate are now implemented in
> thousands, possibly millions of hardware and software products.
Just to be accurate, PKWare did not donate the algorithm, they donated
the format. In my humble opinion, well documented, open formats are
more important than open source, since open specifications foster
competition as opposed to monopolistic behavior. In general, the
former is better for consumers than the latter.
As an aside, PKWare patented a particular algorithm for LZ77 formats.
That however did not impact the adoption of the format, since there
are good ways to deflate without infringing on that patent, or other
LZ77-relevant patents.
Mark Adler
Stuart Caie
> Just to be accurate, PKWare did not donate the algorithm, they donated
> the format.
I stand corrected.
> In my humble opinion, well documented, open formats are
> more important than open source, since open specifications foster
> competition as opposed to monopolistic behavior. In general, the
> former is better for consumers than the latter.
While I agree with you in the context of fostering adoption of a format for
interchange, my main point in mentioning zlib is that many software products
would not feature compression at all, would feature poor or unreliable
compression, or would use commercial compression libraries if it were not
for free and easy to use libraries like zlib. Compression makes it possible
to reduce media/bandwidth/hardware costs, and a free and ready to use,
simple to integrate compression library gives the greatest cost saving in
that regard.
I acknowledge that the key factor in adoption of the ZIP file format for
data interchange, other than its existing popularity, is its open format
specification rather than any one implementation of the specification.
Regards
Stuart
Sachin Garg
"Mark Adler" <mad...@alumni.caltech.edu> wrote in message
news:7ab39013.03072...@posting.google.com...
> Now however, PKWare appears to want to make parts of the format a
> trade secret, which as you point out completely undermines what makes
> the .zip format useful in the first place. In addition to the
> encryption, they have also declined to document the deflate64 format
> in their application note, despite at least two revisions of that note
> since deflate64 was introduced. In this case, it turns out to be not
> very difficult to reverse engineer the format. However the corporate
> intent is clear. The corporate intent is also self-destructive.
>
> So, now may be the time for the community, in particular the community
> that reads this newsgroup, to develop an open, scalable cross-platform
> format that supports archives of directory structures, files, and
> meta-data, high-quality lossless compression, and high-quality
> encryption and authentication. "Cross-platform" does not mean
> "Windows and Mac", but rather as wide a range of platforms as there
> are contributors. The PNG format effort is in my opinion a good model
> for this sort of development. (I played a small part in that
> development.)
It sure is a very good idea.
Having a format created by the community would ensure that it stays with the
community.
Thinking of technical (or legal) expertise, I feel this group has enough of
it, but the noise-to-signal ratio is usually too high over here.
So, if we at comp.ression really want to do something worthwhile we would
need to develop a way of reaching conclusions (which seems to be a tough
job).
How about trying to update the FAQ to start with? It has been showing age
since a long time (just like zip)
Sachin Garg [India]
http://sachingarg.cjb.net
http://sachingarg.go.to
apm
> Darryl Lovato <dlo...@aladdinsys.com> wrote in message news:<dlovato-991D82...@newssvr21-ext.news.prodigy.com>...
> trade secret, which as you point out completely undermines what makes
> the .zip format useful in the first place.
> So, now may be the time for the community, in particular the community
> that reads this newsgroup, to develop an open, scalable cross-platform
> format that supports archives of directory structures, files, and
> meta-data, high-quality lossless compression, and high-quality
> encryption and authentication.
What about gzip?
gpg, used after gzip, adds the encryption and authentication.
Free Software, cross-platform, unencumbered by patents.
All we need now, primarily for the Windoze users, is a
cross platform GUI that glues them together.
-Andrew Marlow.
apm
> released their own variant methods of adding strong encryption to the
> zip standard, which is bad enough - neither is compatible with each
> other, and both are obviously not backward compatible with the thousands
> of zip implementations out there already.
Goodbye .zip, hello .gz
-Andrew M.
Marco Schmidt
[Creating a new, open archive file format]
>It sure is a very good idea.
>
>Having a format created by the community would ensure that it stays with the
>community.
Why are you so certain about that? If the new format became very
popular, what should keep someone from adding a proprietary extension,
as in the case of zip?
>Thinking of technical (or legal) expertise, I feel this group has enough of
>it, but the noise-to-signal ratio is usually too high over here.
The collective expertise here is certainly sufficient, but I'm still
not sure whether creating a new format is such a good idea.
When PNG was designed in 1995, there were clear incentives. The most
prominent one probably was the fact that no such format existed before
(streamable, good compression for most image types, support for
paletted and truecolor data, support for both a transparency index and
an alpha channel, etc.).
I don't see the lack of possibilities on the field of archive formats
and archiver applications today. There are
* free applications for those who do not need to have the bleeding
edge (regarding both speed and compression ratio), including
"old-style ZIP", tar + gzip / bzip2 (although the latter is pretty
good),
* commercial applications with proprietary formats (RAR, SITX,
"new-style ZIP", ...) which are faster and compress better for those
who can spend more money and need the improvements or extra features
and
* specialized applications like rsync that do a better job at some of
the tasks that archivers are / were used for (here: synchronizing data
on different machines).
Even if this group succeeded in
* providing most or all of the features of commercial archivers,
* tuning the code for speed and
* making it run on most platforms,
there is the problem of others adding proprietary extensions to it.
Besides, there is the question of what to do when new compression
methods arise that make a significant improvement, maybe (probably)
only for one type of data. Will the community add it and render older
applications incompatible? As far as I know it is unlikely that new
compression types will ever be added to PNG for this reason, the
(justified, in my opinion) fear of making the format less useful.
Another problem of a less technical nature will arise. In order to
compete with proprietary archivers successfully, most of their
features would have to be replicated. This means a nice GUI interface,
a good online help and various other small features that your average
engineers don't consider important (because they don't need them
themselves) have to be implemented. These tasks often do not get done
in open projects because they are boring.
Who's going to make sure that people will be able to mail an archived
version of their spreadsheet directly from inside Excel? Who will test
full drag and drop compliance on Windows, Mac OS, Linux with KDE,
Linux with Gnome, etc.?
If these things do not get implemented, the format will remain
interesting only for those who use it for ideological reasons
(openness above all) or who are tech-savvy enough to know about its
core qualities. And the latter group will only switch if there are
significant advantages over, say, tar + bzip2, which is not that easy
to accomplish.
The program would also need to support the various popular existing
file formats, because those won't vanish over night. Any end user
deciding for archiver software will pick the one that can handle most
of the already annoying flood of different formats.
Now that I've listed necessary features another problem becomes
visible - the program / project would have to do many things at once.
Most of the time, open projects succeed only if they do one thing, and
that very good.
These are the reasons why I'm pessimistic. Although this would
certainly be an interesting project.
Regards,
Marco
Darryl Lovato
ap...@student.open.ac.uk (apm) wrote:
There is a lot more to creating an modern archive format than simply
glueing a tar, pgp, and gzip together.
- Darryl
Mark Adler
> What about gzip?
> gpg, used after gzip, adds the encryption and authentication.
gzip only provides one aspect, compression. Furthermore, if
compression is more important than speed and memory, then there are
more modern compression methods than gzip (or bzip2 for that matter)
that should be available in a complete tool along with the gzip
method.
The missing piece in your proposal is archiving. This is provided on
Unix by tar (or gnutar), on Windows by zip (which also includes
compression), and on Macs by disk image or StuffIt. What is needed in
addition to compression and encryption is an open specification of a
universal archive format that provides both identical restoration
within a single platform, and best efforts restoration across
platforms. It should handle Unix, Windows, Mac, and probably several
others, such as BEOS and Palm and maybe some weird IBM machines
(OS/390, AS/400).
GnuPG is indeed what should be used for the encryption/authentication
component.
mark
Stuart Caie
> There is a lot more to creating an modern archive format than simply
> glueing a tar, pgp, and gzip together.
There's not much. Most of archive layout is common sense. There have been a
few innovations, but most advances are simply a reflection of what the users
want. To engineer these requirements isn't too hard; to think them up, or
implement them well, is.
Some examples just from the top of my head:
- extensible, logical, machine parsable file format (like the Electronic
Arts IFF specification from 1985)
- possible to add to archive just by appending
- 64 bit (or larger) offsets, lengths and counts
- unicode filepaths; unlimited length filepaths
- solid archiving (more than one file in the same compressed stream)
- sorting files by datatype before solid compression
(so similar files are near to one another in the compression stream)
- compressed file headers to save more space.
- recognising specific file formats directly (e.g. bitmap pictures,
16 bit sound samples, executables) and selecting the best preprocessing or
compression mode for them.
- using indicies (a trie, for example) for seeking in archives with very
large numbers of files, rather than reading all the file headers.
- storing yet more metadata in the headers (e.g. a preview of a picture, the
icon of an executable or the first paragraph of a text file), especially
for filesystem-like browsing of the archive without unpacking much.
- recovery files to reconstitute missing parts of multi-part split archives
Personally, I feel most innovation has gone into A/V container formats
recently (e.g. Quicktime or Matroska), rather than archive headers. But some
ideas have been truly innovative, such as RAR's "script" which allows a big
list of different spans with different processing and compression methods,
rather than a single compression method. For example, the archiver could
recognise base64 encoding when compressing XML, and could mark that span in
the script as being base64 encoded. It could then decode that and pass it
directly as binary through the compressor.
Regards
Stuart
Sachin Garg
"Marco Schmidt" <marcos...@geocities.com> wrote in message
news:iu67ivst6pjtbvnuc...@4ax.com...
> Sachin Garg:
>
> [Creating a new, open archive file format]
>
> >It sure is a very good idea.
> >
> These are the reasons why I'm pessimistic. Although this would
> certainly be an interesting project.
>
> Regards,
> Marco
Well, I think you may be right...
Especially, given the number of already available file formats, it might not
help adding another one to list.
Sachin Garg [India]
Darryl Lovato
Stuart Caie <ky...@4u.net> wrote:
You just described the StuffIt ".sitx" file format :-) You missed a
couple things though - built-in text encoding (base 64, base 222, etc),
and encryption (blowfish, aes, des, etc).
It's already been done - a modern file format exists already - the
question is how to get .zip users to switch to it.
> Personally, I feel most innovation has gone into A/V container formats
> recently (e.g. Quicktime or Matroska), rather than archive headers.
With the exception of the stuffit format, I agree.
> But some
> ideas have been truly innovative, such as RAR's "script" which allows a big
> list of different spans with different processing and compression methods,
> rather than a single compression method. For example, the archiver could
> recognise base64 encoding when compressing XML, and could mark that span in
> the script as being base64 encoded. It could then decode that and pass it
> directly as binary through the compressor.
All this could be added on at a later date, if the architecture is built
to handle it - which .sitx is.
- Darryl
> Regards
> Stuart
>
Kevin Easton
[...]
> - recovery files to reconstitute missing parts of multi-part split archives
(This is a bit of a strange topic for comp.compression, being about
adding redundancy rather than removing it, but oh well..)
There's no reason why there shouldn't be a user-controllable level of
redundancy put into the output, even as a single file - something
smarter than a simple parity, more like Reed-Solomon or similar.
- Kevin.
Eduardo
> In article <3f21...@news.intplsrv.net>,
> "John Smith" <som...@somewhere.com> wrote:
>
> > Perhaps Aladdin should document the sitx format and open it up?
>
> Perhaps, but every time we've looked at that option in the past (and we
> do regularly) the upside is outweighed by the downside. If you don't
> think there are any downsides to doing open source - just look at what
> happened to PKWare and what's going on with ".zip" right now.
>
Well, there are downsides if you do the same PKWare did. Why don't
look succesfull examples, as SGI with OpenGL, the different internet
protocols, ftp, http, nntp, pop, or even the transport/identification
protocols (tcp,udp,ip,icmp,rtp), the main C language... all of them
are good examples with one thing in common, there is behind them, a
powerfull muppet-hand that controls them. PKZip didn't maintain that
leadership.
*******************************************************************************
This document represent my ideas. They are original from me. It's
forbidden think the same than me, without previous payment. If you
agree me, PAY. If you don't do so, you are infringing the copyrigth
laws. If you divulge my ideas in any media, you will be hunt, skin and
cook.
Camping Gaz
if zip is dead (i won't disagree, it's old enough) what is the alternative
so that we ALL start using it?
"[ Doc Jeff ]" <m...@privacy.net>
news:8k53iv480vpaupm7s...@4ax.com...
> On Fri, 25 Jul 2003 17:00:58 GMT, Darryl Lovato
> <dlo...@aladdinsys.com> wrote:
>
> >file format is now irrevicably fractured, and it's sole benefit
> >(umbiquity) has been lost forever.
>
> irrecovably, and I suspect you meant ubiquity which won't change. zip
> files are all over the place as it is.
Darryl Lovato
"Camping Gaz" <p...@wanadoo.fr> wrote:
> irrevocably then.
>
> if zip is dead (i won't disagree, it's old enough) what is the alternative
> so that we ALL start using it?
.SITX of course :-)
It has everything that you would want from a modern archiver/format:
* State of the Art compression, and more improvements to come.
* State of the Art encryption, updated as needed.
* Built-in error correction - archives can "heal" themselves from common
damage.
* Built-in text encoding (base 64, 222, etc)
* Designed with all current file systems in mind, and extensible as new
ones appear. Optimized for single and multi-fork file systems,
automatically handles create/mod date conversion, etc.
* Block Mode
* File Optimizer/Filters - file-type specific compression
* Multi-lingual support.
* Modern code base
* Excellent User interface, customer & tech support
* Developer libraries available on Mac, Win, Unix so far
* 15 year old Public software company (only publicly traded
compression/archiving company) backing the product
* History of technology improvements - it won't fall behind like zip did
* Already the standard on the Mac platform, rapidly gaining marketshare
on other platforms.
* supports all existing file formats (zip, rar, etc, etc)
* Free decompressor, shareware compressors, and full commercial products
available on all platforms.
I'm sure there is more, but it's getting late, and I think this is
enough of an "advertisement" of my companies product.
- Darryl
Heike West
found a bug. StuffIt didn't compress folders who had a french accent
aigu in their name when i selected a couple of folders and compressed
through the context menu in the Windows Explorer.
Actually StuffIt should have higher market shares in the Windows world
as it connects Mac & Win. I think one reason for this is that StuffIt
looks so "bloated". All this plugins for Office and the archive
explorer and what not. Why don't you make all this optional during the
installation? Even better offer a slim version for $ 20 or so which
has not more options than WinRAR. I seriously doubt anybody except
your programmers ever used all this things. Most people work through
the Windows Explorer and there is not even an Icon for StuffIt in the
context menu so that you can find it more quickly. But the option in
the context menu to compress all selected folders into seperate
archives is superb! I wonder why no other program has this option.
Darryl Lovato <dlo...@aladdinsys.com> wrote in message news:<dlovato-0B4A95...@newssvr13-ext.news.prodigy.com>...
Thomas Richter
> In article <bgsjbc$d5m$1...@news-reader5.wanadoo.fr>,
> "Camping Gaz" <p...@wanadoo.fr> wrote:
>
>
>>irrevocably then.
>>
>>if zip is dead (i won't disagree, it's old enough) what is the alternative
>>so that we ALL start using it?
>
>
> .SITX of course :-)
>
> It has everything that you would want from a modern archiver/format:
Does it have a port into the Linux world I can use as "free software" in
the same sense I'm able to use "zip" and "unzip" from my command line? (-;
So long,
Thomas
Darryl Lovato
Thomas Richter <th...@math.tu-berlin.de> wrote:
Go to this link to download the linix version:
<http://www.stuffit.com/unix/index.html>
- Darryl
> So long,
> Thomas
>
Darryl Lovato
heike...@gmx.net (Heike West) wrote:
> Darryl - I've tested your StuffIt for Windows a few weeks ago and
> found a bug. StuffIt didn't compress folders who had a french accent
> aigu in their name when i selected a couple of folders and compressed
> through the context menu in the Windows Explorer.
I'll check into this.
> Actually StuffIt should have higher market shares in the Windows world
> as it connects Mac & Win. I think one reason for this is that StuffIt
> looks so "bloated". All this plugins for Office and the archive
> explorer and what not. Why don't you make all this optional during the
> installation?
Good suggestion. I agree that there is a lot of stuff in the product -
it's more than just a "basic" archiver.
> Even better offer a slim version for $ 20 or so which
> has not more options than WinRAR. I seriously doubt anybody except
> your programmers ever used all this things.
Most people will just remove the parts they don't use after working with
the program for a few weeks.
> Most people work through
> the Windows Explorer and there is not even an Icon for StuffIt in the
> context menu so that you can find it more quickly. But the option in
> the context menu to compress all selected folders into seperate
> archives is superb! I wonder why no other program has this option.
>
Thanks!
- Darryl
Marco Schmidt
[...]
>* Free decompressor, shareware compressors, and full commercial products
>available on all platforms.
No, they're not.
Besides Windows and Mac you offer Solaris / Sparc, Solaris / Intel and
Linux (probably Intel only). That's already a nice list, but hardly
"all platforms". Check out unzip's platform availability:
<http://www.info-zip.org/pub/infozip/UnZip.html>. I realize that it's
not profitable for a business to support that many platforms. But then
don't claim to support all of them.
That's exactly why I think nothing comes close to zip or tar in terms
of convenience. Programs for those formats are available (almost)
everywhere.
Regards,
Marco
Darryl Lovato
Marco Schmidt <marcos...@geocities.com> wrote:
> Darryl Lovato:
>
> [...]
>
> >* Free decompressor, shareware compressors, and full commercial products
> >available on all platforms.
>
> No, they're not.
>
> Besides Windows and Mac you offer Solaris / Sparc, Solaris / Intel and
> Linux (probably Intel only). That's already a nice list, but hardly
> "all platforms".
OK, I should have said all "main" platforms, or all "common platforms"
I believe that we cover 95+% of all computer users with the list of
supported operating systems we have.
> Check out unzip's platform availability:
> <http://www.info-zip.org/pub/infozip/UnZip.html>. I realize that it's
> not profitable for a business to support that many platforms. But then
> don't claim to support all of them.
Fair enough. We try to port to as many platforms as possible. So long
as we can just barely break-even, we'll try to port it.
> That's exactly why I think nothing comes close to zip or tar in terms
> of convenience. Programs for those formats are available (almost)
> everywhere.
Which is why I started this thread in the first place - This is no
longer the case. There are now different variations of zip, and they
most certainly haven't been ported to other platforms, and as far as I
know, they are not even supported by any other program (other than the
author of the variant) on the common platforms.
Aladdin is still trying to support the PKware variant, and have not been
able to do it with what is currently published - let alone the whole
patent issue that PKWare threw into the mix...
- Darryl
> Regards,
> Marco
Dirk Stoecker
> .SITX of course :-)
Someone from Alladin themselfs. Nice.
> It has everything that you would want from a modern archiver/format:
>
> * State of the Art compression, and more improvements to come.
> * State of the Art encryption, updated as needed.
> * Built-in error correction - archives can "heal" themselves from common
> damage.
> * Built-in text encoding (base 64, 222, etc)
> * Designed with all current file systems in mind, and extensible as new
> ones appear. Optimized for single and multi-fork file systems,
> automatically handles create/mod date conversion, etc.
It does surely not support Amiga filesystems. At least the prvious
versions did not and I do not think anyone improved that.
> * Block Mode
> * File Optimizer/Filters - file-type specific compression
> * Multi-lingual support.
> * Modern code base
Uih. Was there a big rework between StuffIt 6 and the newest version? At
least the decrunch code in the binaries of StuffIt 6 looked not so
modern. There have been many possible speed improvements.
> * Excellent User interface, customer & tech support
> * Developer libraries available on Mac, Win, Unix so far
> * 15 year old Public software company (only publicly traded
> compression/archiving company) backing the product
Which started to sell StuffIt with unlicensed compression algorithms.
> * History of technology improvements - it won't fall behind like zip did
> * Already the standard on the Mac platform, rapidly gaining marketshare
> on other platforms.
> * supports all existing file formats (zip, rar, etc, etc)
Have a look at http://www.dstoecker.de/xadclients.html and tell me how
many of the formats supported there you also support. Do NOT tell me you
support ALL existing file formats. Maybe you support the majority of
common formats nowadays, but not all of the THOUSANDS existing compression
formats.
> * Free decompressor, shareware compressors, and full commercial products
> available on all platforms.
No. You only support a minor subset of plattforms.
> I'm sure there is more, but it's getting late, and I think this is
> enough of an "advertisement" of my companies product.
So some questions, you may answer:
1) Is there an open-source variant of the StuffIt own's archiver formats
(including the encryption formats). I do not ask for the compression
only the decompression. Can I port such tool (probably called unstuff)
to any other plattform myself?
2) Would your company offer me free access to the decryption/dearchiving
formats and algorithms (at least descriptions) for the current and the
older formats of StuffIt? Without NDA?
I think not, that you will answer any of the two questions with yes. Sorry
StuffIt X is no replacement for zip. It will never be as long as it is
closed copyrighted source.
The Amiga learned to unarchive StuffIt 6 about one and a half year ago and
today the newest StuffIt can not be dearchived due to your StuffIt X which
again introduced changes in file-formats and algorithms. Support for some
older formats has been dropped from your expander software, so I cannot
dearchive older files. There are no open sources, so nobody can make an
own decompressor if your company chooses to die or stop StuffIt. There is
no useful possibility to recover old data which uses StuffIt compression
some years in the future.
I'm waiting for the format specifications to update my Amiga decompressor.
Ciao
--
____ _ _ ____ _ _ _ _ ____
| | | | | | \ / | | | the cool Gremlin from Bischofswerda
| __ | ____| | \/ | | | WWW: http://www.dstoecker.de/
| | | | | | | | PGP key available on www page.
|____| _|_ |____| _|_ _|_ |____| I hope AMIGA never stops making fun!
Marco Schmidt
>Which is why I started this thread in the first place - This is no
>longer the case. There are now different variations of zip, and they
>most certainly haven't been ported to other platforms, and as far as I
>know, they are not even supported by any other program (other than the
>author of the variant) on the common platforms.
I was a bit unclear on that - I meant zip files that I create.
Everybody will be able to read those because I'm not going to use the
extensions. And that way I don't have to explain (potentially
non-computer-savvy) people to install some particular software just to
extract my archive.
>Aladdin is still trying to support the PKware variant, and have not been
>able to do it with what is currently published - let alone the whole
>patent issue that PKWare threw into the mix...
Yes, that is annoying, but you keep your product under tight control
just as well. I understand the reasons as you explained them earlier
in this thread. I like the product and appreciate the work that has
gone into it, but:
- Those five percent of people using an unsupported platform are
important to me.
- I'd like to be able to easily access an archive on a CD-R 15 years
from now (well, if the CD-R survives). Not sure that SITX-reading
software will make it, but I'm quite certain about (the old) ZIP
format.
Regards,
Marco
CD
Daryl,
I gave that a try.
The first thing I noticed was the claim on the page 'Nothing
compresses files smaller than StuffIt - nothing!' That sort of claim
rings alarm bells, and would have put me off immediately if I'd just
wandered there rather than been looking for it specifically. A few
less exclaimation marks as well please.
My next concern was that the html readme file didn't explain all the
options, although the manual pages covered what I was looking for. The
install file needs a little work - does one really need to put the
directory in $PATH to be able to use it? Isn't it a little pointless
to explain how to untar in a file contained in a tar?
I gave the program a try. It didn't compress as well as my usual
choice for either of the first two samples I tried - yep, the
overblown claim bit the dust - but was slightly better on a log file.
There didn't seem to be much in the way of the archiving options I'd
like. The comments in this thread and descriptions on the website for
'deluxe' variants suggest the version I tried is missing a lot of the
ones Stuffit does have in other incarnations. Why? Isn't a a simple
job to recompile for a different platform?
So, what's available for the platform I'm using just now is only
reasonable. What you have for some platforms may be quite good apart
from compression ratio - although many people will say that's the most
important aspect - but isn't free in either sense. I don't see any
reason to change from the zip, etc. formats if I want widely
understandable archives even if the extension may be overloaded; what
can you offer to convince us otherwise?
C.David
Darryl Lovato
cda...@euromail.sk (CD) wrote:
> Darryl Lovato <dlo...@aladdinsys.com> wrote in message
> news:<dlovato-545B04...@news.sf.sbcglobal.net>...
> > In article <3F340999...@math.tu-berlin.de>,
> > Thomas Richter <th...@math.tu-berlin.de> wrote:
> >
> > > Darryl Lovato wrote:
>
> > > Does it have a port into the Linux world I can use as "free software" in
> > > the same sense I'm able to use "zip" and "unzip" from my command line?
> > > (-;
> >
> > Go to this link to download the linix version:
> >
> > <http://www.stuffit.com/unix/index.html>
> >
> > - Darryl
>
> Daryl,
>
> I gave that a try.
>
> The first thing I noticed was the claim on the page 'Nothing
> compresses files smaller than StuffIt - nothing!' That sort of claim
> rings alarm bells, and would have put me off immediately if I'd just
> wandered there rather than been looking for it specifically. A few
> less exclaimation marks as well please.
Yea, our marketing department doesn't like to "count" research
compressors, etc. if it's not a "product", they don't wan't to count it
- which makes sense, and is true of what most "consumer" use, know
about, etc - but might not be the best wording for people on this
newsgroup.
> My next concern was that the html readme file didn't explain all the
> options, although the manual pages covered what I was looking for. The
> install file needs a little work - does one really need to put the
> directory in $PATH to be able to use it? Isn't it a little pointless
> to explain how to untar in a file contained in a tar?
I'll pass these comments onto the product manager.
> I gave the program a try. It didn't compress as well as my usual
> choice for either of the first two samples I tried - yep, the
> overblown claim bit the dust - but was slightly better on a log file.
What did you compare it against? Also, what format and compression
setting did you use in StuffIt?
> There didn't seem to be much in the way of the archiving options I'd
> like. The comments in this thread and descriptions on the website for
> 'deluxe' variants suggest the version I tried is missing a lot of the
> ones Stuffit does have in other incarnations. Why? Isn't a a simple
> job to recompile for a different platform?
The core "compression engine" is a simple re-compile, and is designed to
be multi-platform, but it isn't simple porting the rest of it - the U/I
for example.
> So, what's available for the platform I'm using just now is only
> reasonable. What you have for some platforms may be quite good apart
> from compression ratio - although many people will say that's the most
> important aspect - but isn't free in either sense. I don't see any
> reason to change from the zip, etc. formats if I want widely
> understandable archives even if the extension may be overloaded; what
> can you offer to convince us otherwise?
Today, I agree - but the same could be said of the zip format when it
first started taking over from Arc. There will become a critical mass,
at which point "enough" potantial target users of the files you send,
that you won't have a problem sending a .sitx file. In the mean time,
you can use the StuffIt product to create zip files (slightly smaller,
AND compatible with PK/WinZip/Etc) for sending to others, and use .sitx
for compression on your own machine - when the file isn't going to be
sent anywhere.
- Darryl
> C.David
Darryl Lovato
Dirk Stoecker <dsto...@gmx.de> wrote:
> On Thu, 7 Aug 2003, Darryl Lovato wrote:
>
> > .SITX of course :-)
>
> Someone from Alladin themselfs. Nice.
>
> > It has everything that you would want from a modern archiver/format:
> >
> > * State of the Art compression, and more improvements to come.
> > * State of the Art encryption, updated as needed.
> > * Built-in error correction - archives can "heal" themselves from common
> > damage.
> > * Built-in text encoding (base 64, 222, etc)
> > * Designed with all current file systems in mind, and extensible as new
> > ones appear. Optimized for single and multi-fork file systems,
> > automatically handles create/mod date conversion, etc.
>
> It does surely not support Amiga filesystems. At least the prvious
> versions did not and I do not think anyone improved that.
No it doesn't. I think you may be the only person in the last year that
has asked for it. Does anyone still use the Amiga?
> > * Block Mode
> > * File Optimizer/Filters - file-type specific compression
> > * Multi-lingual support.
> > * Modern code base
>
> Uih. Was there a big rework between StuffIt 6 and the newest version? At
> least the decrunch code in the binaries of StuffIt 6 looked not so
> modern. There have been many possible speed improvements.
Stuffit 7 was when we introduced the full re-write and new .sitx format
- it was about a year ago.
> > * Excellent User interface, customer & tech support
> > * Developer libraries available on Mac, Win, Unix so far
> > * 15 year old Public software company (only publicly traded
> > compression/archiving company) backing the product
>
> Which started to sell StuffIt with unlicensed compression algorithms.
>
> > * History of technology improvements - it won't fall behind like zip did
> > * Already the standard on the Mac platform, rapidly gaining marketshare
> > on other platforms.
> > * supports all existing file formats (zip, rar, etc, etc)
>
> Have a look at http://www.dstoecker.de/xadclients.html and tell me how
> many of the formats supported there you also support. Do NOT tell me you
> support ALL existing file formats. Maybe you support the majority of
> common formats nowadays, but not all of the THOUSANDS existing compression
> formats.
True, and that's an impressive list on that web page. But... Does
anyone really care about decompressing "NiteTimeGamesDos FS" or "Unreal
package". It's impossible to support everything.
> > * Free decompressor, shareware compressors, and full commercial products
> > available on all platforms.
>
> No. You only support a minor subset of plattforms.
We support 95+% of the platforms people us. I don't think I've heard a
single person ask us to compile a version for the latest Cray
Supercomputer. If you count pc/windows and something like a cray as the
same, they you are technically correct, but that's not the real world.
> > I'm sure there is more, but it's getting late, and I think this is
> > enough of an "advertisement" of my companies product.
>
> So some questions, you may answer:
>
> 1) Is there an open-source variant of the StuffIt own's archiver formats
> (including the encryption formats). I do not ask for the compression
> only the decompression. Can I port such tool (probably called unstuff)
> to any other plattform myself?
No to the first part - yes to the second part - under a very strict
license - as I've said before - I'm would like to port .sitx to as many
platforms as possible - and we realize we can't do it all ourselves -
BUT, we also are not into giving away our hard work, nor are we
interested in getting stuffit into the same messed up position that Zip
is in today, with a million embedded, different implementations, lots of
variations (like what PKWare/WinZip are doing), etc.
> 2) Would your company offer me free access to the decryption/dearchiving
> formats and algorithms (at least descriptions) for the current and the
> older formats of StuffIt? Without NDA?
No. With license/NDA, Yes.
> I think not, that you will answer any of the two questions with yes. Sorry
> StuffIt X is no replacement for zip. It will never be as long as it is
> closed copyrighted source.
We'll see.
> The Amiga learned to unarchive StuffIt 6 about one and a half year ago. And
> today the newest StuffIt can not be dearchived due to your StuffIt X which
> again introduced changes in file-formats and algorithms.
Yep - we keep working on making the product better - putting in new
technology, and upgrading the product. p.s. We don't remember licensing
sit5 format for a Amiga port...
> Support for some
> older formats has been dropped from your expander software, so I cannot
> dearchive older files.
Which ones?
> There are no open sources, so nobody can make an
> own decompressor if your company chooses to die or stop StuffIt.
No, but source code escrow agreements can take care of this unlikely
case.
> There is
> no useful possibility to recover old data which uses StuffIt compression
> some years in the future.
Not true.
- Darryl
Darryl Lovato
Marco Schmidt <marcos...@geocities.com> wrote:
> Darryl Lovato:
>
> >Which is why I started this thread in the first place - This is no
> >longer the case. There are now different variations of zip, and they
> >most certainly haven't been ported to other platforms, and as far as I
> >know, they are not even supported by any other program (other than the
> >author of the variant) on the common platforms.
>
> I was a bit unclear on that - I meant zip files that I create.
> Everybody will be able to read those because I'm not going to use the
> extensions. And that way I don't have to explain (potentially
> non-computer-savvy) people to install some particular software just to
> extract my archive.
Yep. But you don't control what people are going to send you, and if
you are just using it on your own machine, you might as well use sitx
and save the extra disk space. If you are set on using Zip, that's fine
too - just use a stuffit product to do it :-) We compress a couple
percent smaller (on avarage) than PKZip/WinZip and STILL produce
archives that are 100% compatible with the "old" zip standard.
> >Aladdin is still trying to support the PKware variant, and have not been
> >able to do it with what is currently published - let alone the whole
> >patent issue that PKWare threw into the mix...
>
> Yes, that is annoying, but you keep your product under tight control
> just as well. I understand the reasons as you explained them earlier
> in this thread. I like the product and appreciate the work that has
> gone into it, but:
>
> - Those five percent of people using an unsupported platform are
> important to me.
What platforms don't we currently support that you think we should?
> - I'd like to be able to easily access an archive on a CD-R 15 years
> from now (well, if the CD-R survives). Not sure that SITX-reading
> software will make it, but I'm quite certain about (the old) ZIP
> format.
StuffIt will be around. :-)
- Darryl
> Regards,
> Marco
Darryl Lovato
Darryl Lovato <dlo...@aladdinsys.com> wrote:
> In article <f4dejvgvnmbofg50b...@4ax.com>,
> Marco Schmidt <marcos...@geocities.com> wrote:
>
<snip>
> > - Those five percent of people using an unsupported platform are
> > important to me.
>
> What platforms don't we currently support that you think we should?
Here is the list of file formats we currently support on which
platforms...
<http://www.aladdinsys.com/support/techsupport/qanda.php?id=282>
We are currently planning which other formats to support, and what
platforms to port StuffIt to next, so now is a good time for your
feedback - you can send e-mail directly to me <dlo...@aladdinsys.com>
with your requests, so this thread doesn't become longer than it already
has. I have my tech support guys creating a web form for new
platform/format requests, but it isn't done yet - they have been working
on, and just launched a online user forum on our site:
<http://www.aladdinsys.com/support/techsupport/forum/index.php?c=1>
It generally takes about the same amount of effort to port to platform X
vs platform Y, so the one with the most potential users/requests will be
done first. The same goes for additional file format support.
We can't please everyone, but we try.
Thanks,
- Darryl
Marco Schmidt
>Today, I agree - but the same could be said of the zip format when it
>first started taking over from Arc. There will become a critical mass,
>at which point "enough" potantial target users of the files you send,
>that you won't have a problem sending a .sitx file.
But if I understand correctly what happened - zip gained popularity
because it was significantly better than its competitors. You'll have
a hard time getting that status today.
Later zip stayed popular because there were free zip and unzip
programs for most platforms, from several sources.
Regards,
Marco
Marco Schmidt
[...]
>What platforms don't we currently support that you think we should?
I exchange files with people who run FreeBSD / x86. No idea what
market share it has. But I don't say you should support that. You
should only support what paying customers ask you for...
>StuffIt will be around. :-)
Easy to say now. :)
Regards,
Marco
Raymond Wan
Hi,
On Mon, 11 Aug 2003, Darryl Lovato wrote:
> Yea, our marketing department doesn't like to "count" research
> compressors, etc. if it's not a "product", they don't wan't to count it
> - which makes sense, and is true of what most "consumer" use, know
> about, etc - but might not be the best wording for people on this
> newsgroup.
...
> Today, I agree - but the same could be said of the zip format when it
> first started taking over from Arc. There will become a critical mass,
> at which point "enough" potantial target users of the files you send,
> that you won't have a problem sending a .sitx file.
Out of curiosity...exactly what is it that you do at Aladdin sys?
I mean, what's your position? You seem to be selling it quite well...too
devoted to your work. :) I've heard nothing bad about your product;
which makes me a bit skeptical. If I've heard nothing bad, I get the
impression that you haven't done your homework, rather than me looking at
the next best thing to sliced bread.
> In the mean time,
> you can use the StuffIt product to create zip files (slightly smaller,
> AND compatible with PK/WinZip/Etc) for sending to others, and use .sitx
> for compression on your own machine - when the file isn't going to be
> sent anywhere.
And if I wanted to send that SITX file to someone else, I'd have
to decompress it and recompress it as a zip file? What's the point? I
don't know the bpc difference between the two products, but hard disks,
CDR, etc. are cheap...if it offers a compression ratio that isn't as good
as "research compressors", why should I use the SITX format?
I think the one thing you're forgetting is that average
"consumers" operate their computer with the saying, "if it ain't broke,
don't fix it". I'm sure there's still a lot of people that haven't
upgraded to Windows ME, XP, then 2000 just because it came out. A lot of
people are probably still using 98, some maybe even 95. At least with the
readers of this newsgroup, when we see a compression link, we might go and
download the program to try...but for others, I don't think they'd bother.
Ray
John Smith
> Yep - we keep working on making the product better - putting in new
> technology, and upgrading the product. p.s. We don't remember licensing
> sit5 format for a Amiga port...
>
Actually, it isn't neccessary to license the file format specifications.
Reverse engineering for the purposes of file compatibility is legal, unless
the file format uses one or more patents.
Darryl Lovato
<Pine.OSF.4.10.103081...@cassius.its.unimelb.edu.au>,
Raymond Wan <rw...@student.unimelb.edu.au> wrote:
> Hi,
>
> On Mon, 11 Aug 2003, Darryl Lovato wrote:
> > Yea, our marketing department doesn't like to "count" research
> > compressors, etc. if it's not a "product", they don't wan't to count it
> > - which makes sense, and is true of what most "consumer" use, know
> > about, etc - but might not be the best wording for people on this
> > newsgroup.
> ...
> > Today, I agree - but the same could be said of the zip format when it
> > first started taking over from Arc. There will become a critical mass,
> > at which point "enough" potantial target users of the files you send,
> > that you won't have a problem sending a .sitx file.
>
> Out of curiosity...exactly what is it that you do at Aladdin sys?
Chief Technical Officer :-)
> I mean, what's your position? You seem to be selling it quite well...too
> devoted to your work. :)
StuffIt is my "baby". I've been working on it for about 15 years.
> I've heard nothing bad about your product;
> which makes me a bit skeptical. If I've heard nothing bad, I get the
> impression that you haven't done your homework, rather than me looking at
> the next best thing to sliced bread.
The bad part about it is that not everyone has the decompressor yet,
except on the Mac platform, where just about everyone has it.
> > In the mean time,
> > you can use the StuffIt product to create zip files (slightly smaller,
> > AND compatible with PK/WinZip/Etc) for sending to others, and use .sitx
> > for compression on your own machine - when the file isn't going to be
> > sent anywhere.
>
> And if I wanted to send that SITX file to someone else, I'd have
> to decompress it and recompress it as a zip file?
It depends on if the other person has our free expander. We're getting
tons of downloads every day, but have a long way to go to get
decompression support on every machine. If you are sending files to the
same places a lot, suggest they download our free expander - then you
can start saving 30-40, maybe even 50% of the transmission time (depends
on the file). Obviously both zip and sitx, (or anything else) won't
compress already compressed data, so the "difference" between the two
would be smaller than above.
> What's the point? I
> don't know the bpc difference between the two products, but hard disks,
> CDR, etc. are cheap...if it offers a compression ratio that isn't as good
> as "research compressors", why should I use the SITX format?
it's pretty close to the very best known - and getting better. It's FAR
better than Zip. see the following independent test sites for results:
<www.compression.ca>
<www.maximumcompression.com>
>
> I think the one thing you're forgetting is that average
> "consumers" operate their computer with the saying, "if it ain't broke,
> don't fix it".
Agreed, but I think they are about to find out that Zip is "broke".
> I'm sure there's still a lot of people that haven't
> upgraded to Windows ME, XP, then 2000 just because it came out. A lot of
> people are probably still using 98, some maybe even 95. At least with the
> readers of this newsgroup, when we see a compression link, we might go and
> download the program to try...but for others, I don't think they'd bother.
Understood.
- Darryl
> Ray
>
>
>
Darryl Lovato
Marco Schmidt <marcos...@geocities.com> wrote:
> Darryl Lovato:
>
> >Today, I agree - but the same could be said of the zip format when it
> >first started taking over from Arc. There will become a critical mass,
> >at which point "enough" potantial target users of the files you send,
> >that you won't have a problem sending a .sitx file.
>
> But if I understand correctly what happened - zip gained popularity
> because it was significantly better than its competitors. You'll have
> a hard time getting that status today.
OK - lets look at it that way... Here are all the format "adoptions"
that I know of on Mac/Win/Unix. You can see that SITX has exceeded what
has been needed in the past to trigger a "change" of formats.
Calgary+Canterbury 6,062,277 % %smaller-> (old-new)/old
Windows Platform
----------------
Arc 2,594,756 57.20%
PKZip 1,793,403 70.42% 30.88% <- smaller than Arc
Aladdin's new Zip 1,705,862 71.86% 4.88% <- smaller than PKZip
(fully compatible)
STUFFIT X 1,147,614 81.07% 32.73% <- smaller than Aladdin Zip
36.01% <- smaller than PKZip
Unix Platform
-------------
Compress 2,347,999 61.27%
Gzip 1,796,164 70.37% 23.50% <- smaller than Compress
Bzip 1,470,289 75.75% 18.14% <- smaller than Gzip
STUFFIT X 1,147,614 81.07% 21.95% <- smaller than BZip
Mac Platform
------------
PackIt 3,141,590 48.18%
SIT 1.5.1 2,323,773 61.67% 26.03% <- smaller than PackIt
SIT 1.0 2,130,446 64.86% 8.32% <- smaller than SIT 1.5.1
SIT 2.0 2,097,255 65.40% 1.56% <- smaller than SIT 1.0
Compact Pro 1,956,712 67.72% 6.70% <- smaller than SIT 2.0
SIT 3.0 1,861,813 69.29% 4.85% <- smaller than Compact Pro
SIT 5.0 1,404,488 76.83% 24.56% <- smaller than SIT 3.0
STUFFIT X 1,147,614 81.07% 18.29% <- smaller than SIT 5.0
16.06% <- Average % smaller needed for adoption of new/upgraded
standards in the past (9 times)
> Later zip stayed popular because there were free zip and unzip
> programs for most platforms, from several sources.
We have the free expander on most "popular" platforms already.
> Regards,
> Marco
Mikael Lundqvist
> ...
>>I'm sure there's still a lot of people that haven't
>>upgraded to Windows ME, XP, then 2000 just because it came out. A lot of
>>people are probably still using 98, some maybe even 95. At least with the
>>readers of this newsgroup, when we see a compression link, we might go and
>>download the program to try...but for others, I don't think they'd bother.
>
>
> Understood.
>
bother with these stinking cars somebody invented.
I don't really think there's any significant problems with StuffIt being
closed source since WinZip is that too. But almost everybody using a
Wintel box are also using WinZip, and are not complaining about it
(other than possibly being complicated to use for the inexperienced).
The real problems are
1. Most people are using Wintel computers despite the fact that both
Linux and the Macintosh are vastly superior alternatives.
2. StuffIt is originally a Mac utility.
3. Inertia. People will not use it because what they already have on the
HD is "Good Enough" (TM). (if they even know what a "HD" is)
The only way to compete is to do like Sun, make a deal with the OEMs.
The sitx format generally compress much better than the alternatives
like Bzip2, I've only seen Bzip2 compress better on some images (you can
probably improve the filters).
/Mikael
Dmitry Shkarin
> .SITX of course :-)
Why not RAR, 7-ZIP, ACE, SBC?
RAR is available on many platforms. 7-ZIP is open source project. ACE
and SBC also have some interesting properties.
BTW, using of my PPMII without mentioning of my authorship is not so
fair manner.
Thomas Richter
>> It does surely not support Amiga filesystems. At least the prvious
>> versions did not and I do not think anyone improved that.
> No it doesn't. I think you may be the only person in the last year that
> has asked for it. Does anyone still use the Amiga?
Yes, me. Occasionally. That doesn't prove that it has a market share,
it just proves that there's a difference between "open standards"
everyone is able to implement, and "closed standards". Sure enough, no one
is willing to hand out his knowledge for free, and it clearly doesn't make
sense for you to open it in first place, but you should understand that
for *some* systems and *some* people (not necessarely including me, I'm a
human, not a gnu) this is what makes the difference.
Concering establishing a new standard, you've all my best wishes, I know
the problem (being a "JPEG2000 Guru", an image compression system that has
similar acceptance problems, finding its market niche already occopied by
traditional JPEG), I just want to say that on certain platforms, I've my doubt
whether STX will establish a suitable position, regardless of its compression
quality. For most people, a view percent more or less don't count; for
some people on the platforms I prefer working on (Linux, that is), it counts
whether it is "free software" and they won't use it for that reason.
>> 2) Would your company offer me free access to the decryption/dearchiving
>> formats and algorithms (at least descriptions) for the current and the
>> older formats of StuffIt? Without NDA?
> No. With license/NDA, Yes.
Well, Dirk, what's your problem with an NDA? (-;
>> I think not, that you will answer any of the two questions with yes. Sorry
>> StuffIt X is no replacement for zip. It will never be as long as it is
>> closed copyrighted source.
> We'll see.
I've to agree with Dirk here, but maybe for different reasons. I'm not
the guy who says that all software should be free. Just that it's important
to have open standards, at least for products that intend to have a huge
market share. That doesn't include that a) the standard should be easy to
implement and b) you should specify how encoding has to work.
>> The Amiga learned to unarchive StuffIt 6 about one and a half year ago. And
>> today the newest StuffIt can not be dearchived due to your StuffIt X which
>> again introduced changes in file-formats and algorithms.
> Yep - we keep working on making the product better - putting in new
> technology, and upgrading the product. p.s. We don't remember licensing
> sit5 format for a Amiga port...
Well, I guess you can do one of two things: Accepting the 0.1% user loss and
enforce your licence at the price of possibly getting a bad reputation for
some folks, or accepting that you haven't lost a single penny on a market
you didn't want to support in first place. (-;
>> There is
>> no useful possibility to recover old data which uses StuffIt compression
>> some years in the future.
> Not true.
Well, I wouldn't be so sure about that, but time will tell. After all, still
requiring you to keep all the obsolete STX modifications in a modern compressor
also creates a "software monster" you might possibly want to get rid of some
day in the future. No, I'm not trying to avoid STX at all cost, I'm trying to
be realistic, that's all.
So long,
Thomas
CD
> >
> > I gave that a try.
> >
> > The first thing I noticed was the claim on the page 'Nothing
> > compresses files smaller than StuffIt - nothing!' That sort of claim
> > rings alarm bells, and would have put me off immediately if I'd just
> > wandered there rather than been looking for it specifically. A few
> > less exclaimation marks as well please.
>
> Yea, our marketing department doesn't like to "count" research
> compressors, etc. if it's not a "product", they don't wan't to count it
> - which makes sense, and is true of what most "consumer" use, know
> about, etc - but might not be the best wording for people on this
> newsgroup.
I'd suggest prodding your marketing dept. a bit. False claims such as
that really put off a lot of people. I think that if you're trying to
get another format adopted, you'll need to consider the knowledgable
(do I count as that just because I read this group occasionally?)
people as as much as the "consumer" because we're the ones who would
be suggesting it to others.
> > My next concern was that the html readme file didn't explain all the
> > options, although the manual pages covered what I was looking for. The
> > install file needs a little work - does one really need to put the
> > directory in $PATH to be able to use it? Isn't it a little pointless
> > to explain how to untar in a file contained in a tar?
>
> I'll pass these comments onto the product manager.
Continuous improvement awaaaay...
> > I gave the program a try. It didn't compress as well as my usual
> > choice for either of the first two samples I tried - yep, the
> > overblown claim bit the dust - but was slightly better on a log file.
>
> What did you compare it against? Also, what format and compression
> setting did you use in StuffIt?
Rar - which is a product being sold now, presumably not one of the
'research' ones your marketers were concerned about. The more esoteric
ones I use tend to be better compressors, but not as fully featured in
terms of archiving.
I tried the basic and the more advanced stuff methods, as well as
trying to se how well it worked at zip, etc. Note: using the -formats
option (the instructions said not all compression formats were
accepted, and to use that option to see which were) the program said
it could cope with bzip2, but it didn't. I tried all the compression
levels for your own format (what I could set the level to for any
given compression method was one of the things missing fromt html) -
0, 13 and 15. I ran into warnings about too large files with some of
them.
> > There didn't seem to be much in the way of the archiving options I'd
> > like. The comments in this thread and descriptions on the website for
> > 'deluxe' variants suggest the version I tried is missing a lot of the
> > ones Stuffit does have in other incarnations. Why? Isn't a a simple
> > job to recompile for a different platform?
>
> The core "compression engine" is a simple re-compile, and is designed to
> be multi-platform, but it isn't simple porting the rest of it - the U/I
> for example.
Ah. Is it only available in a GUI or something like that?
> > So, what's available for the platform I'm using just now is only
> > reasonable. What you have for some platforms may be quite good apart
> > from compression ratio - although many people will say that's the most
> > important aspect - but isn't free in either sense. I don't see any
> > reason to change from the zip, etc. formats if I want widely
> > understandable archives even if the extension may be overloaded; what
> > can you offer to convince us otherwise?
>
> Today, I agree - but the same could be said of the zip format when it
> first started taking over from Arc. There will become a critical mass,
> at which point "enough" potantial target users of the files you send,
> that you won't have a problem sending a .sitx file. In the mean time,
> you can use the StuffIt product to create zip files (slightly smaller,
> AND compatible with PK/WinZip/Etc) for sending to others, and use .sitx
> for compression on your own machine - when the file isn't going to be
> sent anywhere.
>
> - Darryl
I thought one of the main reasons PKZip took over was that it usually
offered better compression than Arc, and it's remained because it
became so widespread and has an open spec. I don't completely trust
your assertion that there will become a critical mass for .sitx files
- it doesn't seem to be in a significantly better relative position
than when I first met it however long ago and I don't it as being sure
to improve significantly, and your website's blurb has rather dented
your (that's a corporate 'your') credibility.
Incidentally, I can't use .sitx on all my own machines - MS-DOS's
before v7 don't accept four digit extensions.
C.D.
CD
> In article <Pine.LNX.4.56.03...@kgins2.geo.tu-dresden.de>,
> Dirk Stoecker <dsto...@gmx.de> wrote:
>
> > It does surely not support Amiga filesystems. At least the prvious
> > versions did not and I do not think anyone improved that.
>
> No it doesn't. I think you may be the only person in the last year that
> has asked for it. Does anyone still use the Amiga?
Yes. And more people than that use Amiga filesystems - emulators and
the like. They've broadened their appeal a little, and have started
selling emulator packages.
> > > * History of technology improvements - it won't fall behind like zip did
> > > * Already the standard on the Mac platform, rapidly gaining marketshare
> > > on other platforms.
> > > * supports all existing file formats (zip, rar, etc, etc)
> >
> > Have a look at http://www.dstoecker.de/xadclients.html and tell me how
> > many of the formats supported there you also support. Do NOT tell me you
> > support ALL existing file formats. Maybe you support the majority of
> > common formats nowadays, but not all of the THOUSANDS existing compression
> > formats.
>
> True, and that's an impressive list on that web page. But... Does
> anyone really care about decompressing "NiteTimeGamesDos FS" or "Unreal
> package". It's impossible to support everything.
Well, people chopping chunks of Unreal presumably do. However, I think
the vast majority of people - myself included - wouldn't care that the
program can't deal with rare compression/archiving schemes. The
problems occur when the program is advertised as being able to, but
can't - the minority who bought it for that will be rather annoyed.
> > > * Free decompressor, shareware compressors, and full commercial products
> > > available on all platforms.
> >
> > No. You only support a minor subset of plattforms.
>
> We support 95+% of the platforms people us. I don't think I've heard a
> single person ask us to compile a version for the latest Cray
> Supercomputer. If you count pc/windows and something like a cray as the
> same, they you are technically correct, but that's not the real world.
Ooh! Ooh! That's what I call a target.
We aren't asking you to compile a version for the Crays, because we
can get zip/tar unpackers already and there's no need or desire to be
able to deal with .sitx files on them. As for 'that's not the real
world', we're not about to adopt some format like that we can't deal
with across all such systems.
> No to the first part - yes to the second part - under a very strict
> license - as I've said before - I'm would like to port .sitx to as many
> platforms as possible - and we realize we can't do it all ourselves -
> BUT, we also are not into giving away our hard work, nor are we
> interested in getting stuffit into the same messed up position that Zip
> is in today, with a million embedded, different implementations, lots of
> variations (like what PKWare/WinZip are doing), etc.
If you made suitable code available to people, surely you wouldn't
need to port yourselves. Is this related to my parallel query, where
you say the UI makes porting hard?
C.D.
Stuart Caie
> OK - lets look at it that way... Here are all the format "adoptions"
> that I know of on Mac/Win/Unix. You can see that SITX has exceeded what
> has been needed in the past to trigger a "change" of formats.
An interesting list -- where is RAR and 7-Zip?
From what I've seen, RAR has been adopted as the archiver of choice for
people who care about maximum compression ratio / splitting ability /
missing volume recovery. Similarly, 7-Zip is an open source archiver in the
same sort of market. RAR Labs also hand out the source code to the UNIX
unrar tool, for porting to all the esoteric platforms.
As for StuffIt!, what I have seen is its success over Compact Pro leading it
to becoming the de-facto compression standard on the Macintosh. But it is
also a victim of that niche in other platforms. Windows users would rather
use ZIP or RAR over StuffIt, in the same way they would rather use Windows
Media Player than Quicktime Player -- even though Quicktime may be "better"
in some technical aspect like compression or quality. PC users use Stuffit
to "be compatible" with Macs, when Mac users refuse to use ZIP. This is hard
enough as it is with the Mac's choice of dual-forked files.
Regards
Stuart
Raymond Wan
On Tue, 12 Aug 2003, Darryl Lovato wrote:
> Chief Technical Officer :-)
hehe You *do* know that anyone reading this newsgroup and has a
problem with the program will e-mail you about it, now that we know your
title.
> > I mean, what's your position? You seem to be selling it quite well...too
> > devoted to your work. :)
> StuffIt is my "baby". I've been working on it for about 15 years.
Ah! Well that explains it. If I'd been working on something for
15 years, I'd probably either really hate my job, or really like it.
> The bad part about it is that not everyone has the decompressor yet,
> except on the Mac platform, where just about everyone has it.
Unfortunately, too many people use the Windows/PC package. It's
good that StuffIt at least has a market that you can call your own.
Several compressors are fighting it out on the Windows platform and that's
a tough fight. Have to "win" on your home court before you can move on to
the next one.
> It depends on if the other person has our free expander. We're getting
> tons of downloads every day, but have a long way to go to get
Well, Adobe's reader caught on. I never thought it would, but the
big push for it was not really the average user; but companies that
packaged software with manuals that required the Adobe reader. I
personally hate it...give me back the good ol' days of having a printed
manual; but that's off topic, so I'll digress.
> it's pretty close to the very best known - and getting better. It's FAR
> better than Zip. see the following independent test sites for results:
Yes, the results look impressive, but from the couple of links
I've seen, I at least see winrar and winace -- both aiming for the same
thing as stuffit. That is, have their own formats, while still supporting
zip in the hope that people will switch over.
> > I think the one thing you're forgetting is that average
> > "consumers" operate their computer with the saying, "if it ain't broke,
> > don't fix it".
> Agreed, but I think they are about to find out that Zip is "broke".
So is Windows...so why do we still use it? Or, why do
manufacturers insist on putting it on PCs as OEM? *shrug*
BTW, since your the chief tech officer, one suggestion for
improving your product. Support for (natural) languages. Supporting 95%
of the OS' in the world does not help when the majority of those are in
another language that is different from your program. I saw a French,
German, and Japanese version and that's all -- and from the version
numbers, they look a tad out of date. I don't know anything about Mac's,
but from personal experience, installing English software on a non-English
Windows machine is a no-no. The DLL's and various other things don't
co-operate.
Oh, and here's a good one (sorry all, but this isn't about
compression anymore). I had an Excel file whose filename was in Japanese
that I wanted to open with English Excel ... shouldn't be a problem. I
knew the entire file was in English; but Windows didn't let me open it,
copy it, rename it, erase it...I had to format the disk and lose the data
(I didn't have access to the Japanese PC I made it on anymore).
IMHO, if we're talking about a commercial compressor and not
something for research, the few bits per character won't mean as much as a
localised version with a translated manual, menus, filename support, etc.
I'm sure the translation has been done for Macs, otherwise you couldn't
corner the Mac market...
Ray
Darryl Lovato
Stuart Caie <ky...@4u.net> wrote:
> Darryl Lovato wrote:
> > OK - lets look at it that way... Here are all the format "adoptions"
> > that I know of on Mac/Win/Unix. You can see that SITX has exceeded what
> > has been needed in the past to trigger a "change" of formats.
>
> An interesting list -- where is RAR and 7-Zip?
Rar and 7-Zip didn't get 30% or more marketshare from the existing
"standard" they replaced/tried to replace. My data only showed the "big
adoptions" or "near adoptions". Rar and 7-zip combined probably don't
even have 15% of the windows compression utility market.
> From what I've seen, RAR has been adopted as the archiver of choice for
> people who care about maximum compression ratio / splitting ability /
> missing volume recovery.
Which is 2% of the market, and a bunch of those people are using .sitx
for the same reasons.
> Similarly, 7-Zip is an open source archiver in the
> same sort of market. RAR Labs also hand out the source code to the UNIX
> unrar tool, for porting to all the esoteric platforms.
Source or no source, 7-Zip and RAR are small on the windows platform
compared to .zip. It's true they have some market share on windows - as
does stuffit - almost all the cross platform users, but that's still a
small part of the whole.
> As for StuffIt!, what I have seen is its success over Compact Pro leading it
> to becoming the de-facto compression standard on the Macintosh.
Actually, Compact Pro was a "near takover" - stuffit was the standard
before, Compact Pro came in and took a bunch of the market, then our
next release gained it all back.
> But it is
> also a victim of that niche in other platforms. Windows users would rather
> use ZIP or RAR over StuffIt,
Zip yes - currently :-), but we are pretty close in windows market share
to RAR.
> in the same way they would rather use Windows
> Media Player than Quicktime Player -- even though Quicktime may be "better"
> in some technical aspect like compression or quality. PC users use Stuffit
> to "be compatible" with Macs, when Mac users refuse to use ZIP. This is hard
> enough as it is with the Mac's choice of dual-forked files.
StuffIt does enjoy the bulk of the "cross platform users" The format
was designed to be that way from the start - with zip, rar, etc - other
file systems are more of an afterthought - Mac files are
shoe-horned/hacked into those formats by macbinary'ing the two file
forks together, then adding the macbinary as if it was a regular file.
- Darryl
> Regards
> Stuart
>
Marco Schmidt
[...]
>Windows users would rather
>use ZIP or RAR over StuffIt, in the same way they would rather use Windows
>Media Player than Quicktime Player -- even though Quicktime may be "better"
>in some technical aspect like compression or quality.
Quicktime fails on the Windows platform because the playback is utter
crap. A couple of months ago I couldn't play the large Matrix Reloaded
movie trailer (.mov, 1000 x something pixels) on a modern system
without a dramatic frame drop, which made it unwatchable. The 640 x
something version worked, but it took almost all the CPU. Still no
full screen playback with the 640 version without frame drop.
I could reproduce that behaviour on different modern Intel / Windows
systems.
It's great if those trailers play smoothly on Macs, but that way Apple
is not going to win over the Windows platform. Maybe they don't care
about it.
Regards,
Marco
Marco Schmidt
[...]
>I don't know anything about Mac's,
>but from personal experience, installing English software on a non-English
>Windows machine is a no-no. The DLL's and various other things don't
>co-operate.
I don't experience much problems with English software on my (German)
Windows version. But I rarely install new software.
When I get to choose between German and English, I always pick English
if the software comes from outside of Germany. Translations are often
behind the latest version, as you say, and they sometimes are bad.
People usually can handle "File", "Edit", "Help", etc., but once you
get to "Component subsampling" or "Interpolation type", translations
tend to become either funny or annoying, depending on your mood at the
time. :/
[...]
>IMHO, if we're talking about a commercial compressor and not
>something for research, the few bits per character won't mean as much as a
>localised version with a translated manual, menus, filename support, etc.
That is indeed very important. I remember that there was a plan to
have exactly one Windows version that has all languages built in. That
obviously never happened... I wonder why. Just experienced that
annyoing differentiation yesterday when I downloaded the patch for
that RPC virus. Didn't see the box where you could choose your
language (Microsoft webpages look weird when visited with something
other than IE) and downloaded the wrong version.
[...]
Regards,
Marco
Marco Schmidt
[...]
>Just that it's important
>to have open standards, at least for products that intend to have a huge
>market share. That doesn't include that a) the standard should be easy to
>implement and b) you should specify how encoding has to work.
Sometimes products keep a huge market share /because/ of the lack of
openness. Competitors of a quasi-monopol simply cannot fully
reverse-engineer a format and thus have a disadvantage.
Personally I hope that huge institutions and businesses prefer open
standards because of the longevity of their documents. I yesterday saw
something on CNN about media (not file formats) becoming harder and
harder to access because the necessary hardware isn't built anymore,
and the existing hardware is replaced with newer things. CD-R instead
of 5,25" floppy. All good, unless you have an old medium that you must
read from.
With file formats is less dramatic - mostly there are emulators for
older OSs. But it can be a lot of work to get all the necessary
software to access old data.
Regards,
Marco
Raymond Wan
On Tue, 12 Aug 2003, Marco Schmidt wrote:
> I don't experience much problems with English software on my (German)
> Windows version. But I rarely install new software.
Yes, I'd also expect little problems mixing French and English --
they're from the same language family, after all, and for computers, part
of the same character set (Latin-1, not ASCII... :) ). But there's most
of the Asian languages and Arabic...that's a large part of the world's
population already.
Personally, I'm still waiting for this Unicode-thing that we've
been promised would solve all our language problems. If all these
programs and OS' supposely support Unicode, why did an English program
crash on a Japanese OS (for me)? I suppose on the surface, you can switch
between languages, but the underlying code is still written with on
language in mind. Porting just the menus and the documentation is not
enough... ;)
> People usually can handle "File", "Edit", "Help", etc., but once you
> get to "Component subsampling" or "Interpolation type", translations
> tend to become either funny or annoying, depending on your mood at the
> time. :/
Well, English is my first language, but I feel bad for people who
don't have english as a first language and have to contend with menus the
way Microsoft and other English-speaking companies have designed. I don't
know German, but with Japanese and Chinese, it's funny to see the word
"file" in their native language and then the (F) hotkey next to it. For
english speakers, of course; it makes sense the hotkey for file is f...but
it makes no sense in other languages.
On the bright side, I don't know Japanese but was able to navigate
the File, Edit, and Options menus without reading any of it... :-)
Ray
Dirk Stoecker
> > > * Designed with all current file systems in mind, and extensible as new
> > > ones appear. Optimized for single and multi-fork file systems,
> > > automatically handles create/mod date conversion, etc.
> >
> > It does surely not support Amiga filesystems. At least the prvious
> > versions did not and I do not think anyone improved that.
>
> No it doesn't. I think you may be the only person in the last year that
> has asked for it. Does anyone still use the Amiga?
I really do not care. This was only to show you that your claim is wrong.
You use to much words like "all" when talking about features. That type of
marketing speech often results in comments like above :-) Especially as
usually noone supports Amiga.
> > Uih. Was there a big rework between StuffIt 6 and the newest version? At
> > least the decrunch code in the binaries of StuffIt 6 looked not so
> > modern. There have been many possible speed improvements.
>
> Stuffit 7 was when we introduced the full re-write and new .sitx format
> - it was about a year ago.
I know SITX was introduced some time ago. But I have not so much time to
care for every new compression format. There are too much of them.
> > Have a look at http://www.dstoecker.de/xadclients.html and tell me how
> > many of the formats supported there you also support. Do NOT tell me you
> > support ALL existing file formats. Maybe you support the majority of
> > common formats nowadays, but not all of the THOUSANDS existing compression
> > formats.
>
> True, and that's an impressive list on that web page. But... Does
> anyone really care about decompressing "NiteTimeGamesDos FS" or "Unreal
> package". It's impossible to support everything.
Agreed. But XAD is an open system so it naturally supports the things
users want (and this includes things like "NiteTimeGamesDos FS" as well
:-).
> > > * Free decompressor, shareware compressors, and full commercial products
> > > available on all platforms.
> >
> > No. You only support a minor subset of plattforms.
>
> We support 95+% of the platforms people us. I don't think I've heard a
> single person ask us to compile a version for the latest Cray
> Supercomputer. If you count pc/windows and something like a cray as the
> same, they you are technically correct, but that's not the real world.
But to have a common exchange format, I need the ability to do
uncompression on the most exotic systems (including Cray super computers
and 8 bit machines as well).
> > > I'm sure there is more, but it's getting late, and I think this is
> > > enough of an "advertisement" of my companies product.
> >
> > So some questions, you may answer:
> >
> > 1) Is there an open-source variant of the StuffIt own's archiver formats
> > (including the encryption formats). I do not ask for the compression
> > only the decompression. Can I port such tool (probably called unstuff)
> > to any other plattform myself?
>
> No to the first part - yes to the second part - under a very strict
> license - as I've said before - I'm would like to port .sitx to as many
> platforms as possible - and we realize we can't do it all ourselves -
> BUT, we also are not into giving away our hard work, nor are we
> interested in getting stuffit into the same messed up position that Zip
> is in today, with a million embedded, different implementations, lots of
> variations (like what PKWare/WinZip are doing), etc.
A open-source variant of the decompression stuff would not change your
chances with the compression software, but it would greatly enhance the
usability of the format. See my own XAD system. It does decompression only
to allow interoperability and will never do compression.
Your company will still be the only producer of the compression suite,
but decompression can be ported freely. A big advantage.
> > 2) Would your company offer me free access to the decryption/dearchiving
> > formats and algorithms (at least descriptions) for the current and the
> > older formats of StuffIt? Without NDA?
>
> No. With license/NDA, Yes.
The NDA would prevent me to make an open-source variant. So I need to say
no-thanks. I will stick to reassembling then whenever SITX gets accept by
a larger user scale and it gets important to support it.
> > The Amiga learned to unarchive StuffIt 6 about one and a half year ago. And
> > today the newest StuffIt can not be dearchived due to your StuffIt X which
> > again introduced changes in file-formats and algorithms.
>
> Yep - we keep working on making the product better - putting in new
> technology, and upgrading the product. p.s. We don't remember licensing
> sit5 format for a Amiga port...
You did not. Luckily here in europe we have no software patents at the
moment and also we are allowed to reassemble software to make
interoperability possible.
> > Support for some
> > older formats has been dropped from your expander software, so I cannot
> > dearchive older files.
>
> Which ones?
Lower StuffIt 1 format algorithms (1 to 5 or something like that). Maybe
this was only dropped in the Expanders I tried, but I had problems with
these.
> > There is
> > no useful possibility to recover old data which uses StuffIt compression
> > some years in the future.
>
> Not true.
Which needs to be proven. I had many problems to get the decompression of
some older MacIntosh or Atari formats done when developing certain parts
of XAD. Recreating of an algorithm is much easier, if I have plain text
and crunched text, so run an old compressor is a big bonus to get the
algorithm. You current programs will probably not run on machines build in
10-20 years. They will not run at all on machines in 30-40 years. If you
company drops SITX-support or vanishes and there are no proper emulators
or open sources, there will be big trouble to unarchive files.
And please do not tell me, there will be no interest in stuff archives 40
years ago.
A note at the end: I have nothing against StuffIt. It may be a fine
archiver (although I do not use it), but I want to have a description how
I can get my own data out of an archive. No proprietary software, but a
working description, so I can extract the data under the worst
possibly conditions. Nobody nowadays knows what these worst conditions
will be.
Dirk Stoecker
Hi Thomas.
> >> 2) Would your company offer me free access to the decryption/dearchiving
> >> formats and algorithms (at least descriptions) for the current and the
> >> older formats of StuffIt? Without NDA?
>
> > No. With license/NDA, Yes.
>
> Well, Dirk, what's your problem with an NDA? (-;
It prevents me to make my projects open sources. And hobby development
with little time is hard to do closed source. At least for me.
> >> I think not, that you will answer any of the two questions with yes. Sorry
> >> StuffIt X is no replacement for zip. It will never be as long as it is
> >> closed copyrighted source.
>
> > We'll see.
>
> I've to agree with Dirk here, but maybe for different reasons. I'm not
> the guy who says that all software should be free. Just that it's important
> to have open standards, at least for products that intend to have a huge
> market share. That doesn't include that a) the standard should be easy to
> implement and b) you should specify how encoding has to work.
I have no problem with compression software. I don't want it to be free. I
only want a free and openly documented method to get my data back.
Even Microsoft's CAB-format has a free documentation (with lots of errors
in it :-)
Dirk Stoecker
> >Just that it's important
> >to have open standards, at least for products that intend to have a huge
> >market share. That doesn't include that a) the standard should be easy to
> >implement and b) you should specify how encoding has to work.
>
> Sometimes products keep a huge market share /because/ of the lack of
> openness. Competitors of a quasi-monopol simply cannot fully
> reverse-engineer a format and thus have a disadvantage.
I don't think this is true for compression formats. The main aim of
compression is to keep the files short. So you cannot add redundant data
and other things, which are e.g. implemented in Microsoft word format.
Reverse engineering of compression formats is not the easiest work at all,
but if a company would pay for it, it probably takes about one or two man
months to do so with skilled programmers. It takes longer in homework.
I did that for several older and newer formats and never experienced
bigger troubles.
I think the reputation a compression company can gain if open source
decompression and documentation for the formats exists is worth more than
hiding the formats. BTW, the open source community would do the work to
have a decompressor available on all the target plattforms (saving
manpower in the company to develop the compression technology).
For example all the bigger linux distributors (e.g. SuSE, Debian and
RedHat) would very likely include an unsit tool in their distributions
(like they did with the cabextract tool written by Stuart Caie), so
the Linux market soon gets a nearly 100% coverage for decompression.
Alladdin already improves their compression technology over the time, so
they will always will be in front of their competitors, as these could
only reverse engineer and rebuild. I don't think that releasing the
decompression would speed up the work on the side of competitors
(Especially as the difference in complexity between compression and
decompression algorithms is usually really large).
Marco Schmidt
>I don't think this is true for compression formats. The main aim of
>compression is to keep the files short. So you cannot add redundant data
>and other things, which are e.g. implemented in Microsoft word format.
>Reverse engineering of compression formats is not the easiest work at all,
>but if a company would pay for it, it probably takes about one or two man
>months to do so with skilled programmers. It takes longer in homework.
>I did that for several older and newer formats and never experienced
>bigger troubles.
But can you be sure to have understood all the fields in a file
format's header structure? Maybe rarely used parts were always 0 in
your test files, but in some cases have a different value and a
certain meaning. The designer always has the edge here. Besides, if a
fully committed skilled engineer needs two man months just to
understand the format, it doesn't seem to me that this is an open
source project with a high success chance.
[...]
>For example all the bigger linux distributors (e.g. SuSE, Debian and
>RedHat) would very likely include an unsit tool in their distributions
>(like they did with the cabextract tool written by Stuart Caie), so
>the Linux market soon gets a nearly 100% coverage for decompression.
If the format is really superior, I'm not so sure that nobody will try
to write a compressor. That's probably what Aladdin is afraid of.
On the other hand, with a really superior format people might
reverse-engineer it and (not out of malice, but because of an
incomplete understanding of the format) create corrupted SITX files.
Which is bad for Aladdin because it damages their reputation if a lot
of corrupted SITX files are in the wild. Users will not understand
that Aladdin is not to blame.
>Alladdin already improves their compression technology over the time, so
>they will always will be in front of their competitors, as these could
>only reverse engineer and rebuild. I don't think that releasing the
>decompression would speed up the work on the side of competitors
>(Especially as the difference in complexity between compression and
>decompression algorithms is usually really large).
If I understood Darryl correctly, Aladdin doesn't want its competitors
to gain knowledge of things Aladdin has come up with for the file
format. Once that is in the open, it may be possible to copy that
feature. Coming up with the idea was hard, but implementation is easy.
Regards,
Marco
Dirk Stoecker
> But can you be sure to have understood all the fields in a file
> format's header structure? Maybe rarely used parts were always 0 in
> your test files, but in some cases have a different value and a
> certain meaning. The designer always has the edge here. Besides, if a
I can not and there are always missing fields. The more manpower you have,
the better get the results.
> fully committed skilled engineer needs two man months just to
> understand the format, it doesn't seem to me that this is an open
> source project with a high success chance.
Including the dearchiving algorithms. The formats usually are not that
complicated, thought it seems StuffIt 7 is an completely compressed
format, so you need to find the decompression algorithm first and
afterwards you can check the real file format.
> On the other hand, with a really superior format people might
> reverse-engineer it and (not out of malice, but because of an
> incomplete understanding of the format) create corrupted SITX files.
> Which is bad for Aladdin because it damages their reputation if a lot
> of corrupted SITX files are in the wild. Users will not understand
> that Aladdin is not to blame.
If the format is that superior, it will be reverse engineered anyway. With
or without open format specification and decompression algorithm.
> >Alladdin already improves their compression technology over the time, so
> >they will always will be in front of their competitors, as these could
> >only reverse engineer and rebuild. I don't think that releasing the
> >decompression would speed up the work on the side of competitors
> >(Especially as the difference in complexity between compression and
> >decompression algorithms is usually really large).
>
> If I understood Darryl correctly, Aladdin doesn't want its competitors
> to gain knowledge of things Aladdin has come up with for the file
> format. Once that is in the open, it may be possible to copy that
> feature. Coming up with the idea was hard, but implementation is easy.
I have seen many file formats and also learned a bit about compression
technology at all. I don't think there can be that much exciting news in
the file format itself. The main energy must be in the compression
algorithms.
BTW: Telling how to read a book does not give away the secret how the
book-printing works.
Stuart Caie
> But can you be sure to have understood all the fields in a file
> format's header structure? Maybe rarely used parts were always 0 in
> your test files, but in some cases have a different value and a
> certain meaning. The designer always has the edge here.
Well, it is the aim of a reverse engineer to understand all the fields, so
the job is not finished until every field is understood. The number one
requirement is to have a lot of testing and a lot of test files. Now perhaps
you can understand why it takes a long time, other than the fact that
compressed data streams are very unforgiving of minor differences in algorithm.
You can also determine whether software even sets a field or not. For
example, when I reverse engineered the AHX music format, I discovered 3
bytes in each sample that did not appear to have a meaning. After talking
with the author of AHX, he had renamed three structure members, but
forgotton to delete their predecessors, creating a hole in the structure. A
lot of unknown fields are "reserved for further use" that never get used.
> Besides, if a
> fully committed skilled engineer needs two man months just to
> understand the format, it doesn't seem to me that this is an open
> source project with a high success chance.
How long did it take OpenOffice, WINE, Samba and ScummVM engineers to
understand unknown formats? A very long time. Yet these are all highly
successful projects.
The success of an open source project has nothing to do with the complexity
of the task, and everything to do with the motivation of the developers. The
reason behind the popularity of the above projects is because a lot of users
and developers want access to their data outwith the original "client", even
though it is locked up in a format they do not understand.
> If the format is really superior, I'm not so sure that nobody will try
> to write a compressor. That's probably what Aladdin is afraid of.
This is what brand names and trademarks are for. The difference with ZIP is
that Phil Katz put the term "ZIP" into the public domain, any program can
use the ZIP name themselves, but the term "StuffIt!" is not in the public
domain, and everybody (on the Mac, at least) knows what StuffIt! is and who
makes it. Trademarks can be very strong, for example SEA convinced the
Japanese authors of LhArc to rename to just "LhA" rather than clash with the
trademarked "ARC". I'm sure everyone remembers the ARC vs PKARC litigation.
Also have a look at the Sun v Microsoft java case. Sun revoked Microsoft's
license to use the Java trademark, because they could not make software that
was compatible with the Java open standards.
> Which is bad for Aladdin because it damages their reputation if a lot
> of corrupted SITX files are in the wild. Users will not understand
> that Aladdin is not to blame.
This is an unrealistically elitest view of users. If a user gets a "CRC
error" when unpacking a file, they expect their file is corrupted in some
way, not that their archiver is broken. They then go looking for help from
the archive provider, the archiver author, web boards, Google, etc. If it
turns out that StuffIt! and FooCompress use the SITX format, but are
slightly incompatible, then that rumour will spread around. I'm sure a
company like Aladdin has a PR department that knows how to win that
situation for their benefit and to the detriment of FooCompress.
Open standards can help a company here. The owner of the open standard can
point to the open standard and say "look, here is where FooCompress goes
wrong. Don't use it until it complies with the open standard".
> If I understood Darryl correctly, Aladdin doesn't want its competitors
> to gain knowledge of things Aladdin has come up with for the file
> format.
There's a simple solution to this: never release their software. However, if
they disobey this and actually release software using their SITX file
format, then their competitors will find out what the file format is, as
will everyone else who is motivated to take a look. It's a question of
motivation, not complexity. Nothing can be made secret in software, only
well hidden.
Regards
Stuart
Marco Schmidt
>Well, it is the aim of a reverse engineer to understand all the fields, so
>the job is not finished until every field is understood. The number one
>requirement is to have a lot of testing and a lot of test files. Now perhaps
>you can understand why it takes a long time, other than the fact that
>compressed data streams are very unforgiving of minor differences in algorithm.
I certainly do understand, I have reverse-engineered compressed
formats myself. That's where I got my pessimism from.
>You can also determine whether software even sets a field or not. For
>example, when I reverse engineered the AHX music format, I discovered 3
>bytes in each sample that did not appear to have a meaning. After talking
>with the author of AHX, he had renamed three structure members, but
>forgotton to delete their predecessors, creating a hole in the structure. A
>lot of unknown fields are "reserved for further use" that never get used.
That happens, but being on the other side of things when
reverse-engineering, you don't really know whether you have one of
those useless holes or something which is important in rare cases.
Unless you fully decompile the creator software.
[...]
>How long did it take OpenOffice, WINE, Samba and ScummVM engineers to
>understand unknown formats? A very long time. Yet these are all highly
>successful projects.
>The success of an open source project has nothing to do with the complexity
>of the task, and everything to do with the motivation of the developers. The
>reason behind the popularity of the above projects is because a lot of users
>and developers want access to their data outwith the original "client", even
>though it is locked up in a format they do not understand.
OK, it can happen. I just picture all those aborted, failed projects
in comparison to relatively few successes. Years ago I regularly read
the NTFS Linux driver mailing list, and it was quite frustrating to
see how motivated people can't seem to make progress because of lack
of documentation of that file system.
>This is what brand names and trademarks are for.
[...]
>Also have a look at the Sun v Microsoft java case. Sun revoked Microsoft's
>license to use the Java trademark, because they could not make software that
>was compatible with the Java open standards.
It's hard to get to the point where you really have a brand. Obviously
I can come up with a new archive format and donate it to the Public
Domain, but that doesn't make a difference if nobody is using it. And
you may never get to the point where you have a brand (or you may lose
it if you do) if you tell all your secrets right away.
Having said that, I'm all for open standards. I'm just not optimistic
how these days (with all the competition) one is to get to the top and
at the same time be all open about the format.
>This is an unrealistically elitest view of users. If a user gets a "CRC
>error" when unpacking a file, they expect their file is corrupted in some
>way, not that their archiver is broken. They then go looking for help from
>the archive provider, the archiver author, web boards, Google, etc.
You have users that turn to Google to research error messages in case
of problems? Can I have those? Seriously, from my experience real "end
users" rarely have even the basic abilities to help themselves. Those
help desk stories of people putting floppy disks in CD drives may seem
laughable and exaggerated, but there is a certain truth in them about
the level of expertise of normal users.
>If it
>turns out that StuffIt! and FooCompress use the SITX format, but are
>slightly incompatible, then that rumour will spread around. I'm sure a
>company like Aladdin has a PR department that knows how to win that
>situation for their benefit and to the detriment of FooCompress.
>Open standards can help a company here. The owner of the open standard can
>point to the open standard and say "look, here is where FooCompress goes
>wrong. Don't use it until it complies with the open standard".
I don't think normal users read any sort of computer magazines or
online forums where incompatibilities of different archivers are
discussed.
>There's a simple solution to this: never release their software. However, if
>they disobey this and actually release software using their SITX file
>format, then their competitors will find out what the file format is, as
>will everyone else who is motivated to take a look. It's a question of
>motivation, not complexity. Nothing can be made secret in software, only
>well hidden.
If it's well hidden it may be too costly to reverse engineer it. At
least in a commercial environment where time is money.
The easiest approach for the designer of some format is to not make it
too easy for the competition by not documenting the format.
Regards,
Marco
Dirk Stoecker
> The easiest approach for the designer of some format is to not make it
> too easy for the competition by not documenting the format.
This is the current industry view of the problem. But this does not take
open source into account (a common problem). The software companies of old
style always fight against the open source idea instead arranging with it.
It is not easy to calculate, but in many cases the positive effects of
these open source tools much overwhelm the negative effects. As said, the
benefits of an freely available decompressor would allow a wider range of
computer systems able to handle the file format. And the users may decide
to buy the archiver than. Nowadays everyone uses ZIP, as everyone can
extract zip. If everyone can extract StuffIt, people will also buy the
creator program. It's like with ZIP. People bought the nice and easy to
use WinZIP after ZIP already got widely accepted (and there have always
been freeware competitors which will not exist with StuffIt).
Joe
[ Doc Jeff ] <m...@privacy.net> wrote in message
news:bn34ivolv5qagatg7...@4ax.com...
> On Fri, 25 Jul 2003 21:51:18 GMT, Darryl Lovato
> <dlo...@aladdinsys.com> wrote:
>
> >Aladdin faced a similar issue with the StuffIt format about 5 years ago.
> >We had added-on, then added-on, and then added some more. Although the
> >user features became better, and we kept up with algorithm research, it
> >became too much to support and became clunky.
>
> I think that's exactly where ZIP has wound up. As I said previously, I
> don't even make them anymore.
>
> >The "Zip companies", on the other hand, have done basically nothing
> >(Zip64, and the BZip algorithm addition are small exceptions - which are
> >still not universally supported) to improve Zip compression in the last
> >10+ years, and have focused their efforts on marketing and user
> >interface features - Ignoring the reason people use the product in the
> >first place - to make file's smaller. The Zip format has become
> >static/dead with no real innovation because nobody took ownership to
> >advance the format after PKWare put it into the public domain.
>
> They really haven't done anything of note on the compression side
> since Phil passed away. I have to wonder if he was the only one
> thinking about compression rather than bells & whistles.
>
> >Further, since there are so many unique implementations of the zip
> >format, (hundreds of developers have rolled their own), it would be
> >almost impossible to have a coordinated roll out of a new zip format -
> >some programs would support it, others wouldn't, etc. It would be a
> >real mess for users.
>
> Yes, this would be very bad.
>
> >Maybe Aladdin should have named ".sitx", ".zip2" :-)
>
> Heh. That would've presented an interesting situation.
>
> --
> http://www.cotse.net - Use it, you know you want to.
> If you're too scared to go look for yourself, ask me
> about COTSE. I'd be happy to tell you about it.
I already use RAR/TGZ/etc., just for that reason....
A lot complain about using formats like RAR and OGG, but these are the same
people, who like Microsoft's own format (=controlled) of music, and probally
RealOne too.
It's mostly cause they're too lazy or uninterrested in running something
better.
Andraia Matrix
>
> >Windows users would rather
> >use ZIP or RAR over StuffIt, in the same way they would rather use Windows
You are right... I would rather use ZIP. That guarantees me the
ability to open up that file 10 years from now.
Back in my DOS days, I did play with other archive formats (even ARJ's
"JAR" program) and it's no fun trying to figure out what archiver to
use to access some old data or program.
Most people don't use ZIP as a way of compression any more. We use it
as a way of collecting files into one package. Sort of the "TAR" of
the PC. The compression is just a nice bonus and rarely significant.
> >Media Player than Quicktime Player -- even though Quicktime may be "better"
> >in some technical aspect like compression or quality.
>
> Quicktime fails on the Windows platform because the playback is utter
> crap. A couple of months ago I couldn't play the large Matrix Reloaded
Yeah. The codecs do seem to use a lot of cpu cycles for the quality
it actually gives. And the player itself is very sluggish.
I suspect they've never actually optimized it for the x86
architecture. And certainly not specific P3/P4/Athlon optimizations.
They probably used the same type of "optimizations" for it that they
did when they benchmarked their latest Mac against the latest P4 and
"proved" how much faster their mac was.
> is not going to win over the Windows platform. Maybe they don't care
> about it.
They don't.
But you can't entirely blame Apple.
With only, what, 12-10 million mac users left world wide (Linux has
almost as many desktop users!), and that number dropping daily, Apple
can't really afford to cater to PC users. They've got to concentrate
what little money they have into catering to their paying customers.
If they don't keep them happy, then Apple may not exist at all in 10
years, and wouldn't exist except as a "legacy" company in 5 years.
Supposedly (according to apple), the quicktime media platform is the
most popular in the world, etc. etc. But I suspect their platform
counting and usage counting is of somewhat comparable quality as their
benchmarking ability.
Probably something like how Real & FLASH is... A lot of people have it
installed because it came pre-installed or some program automatically
installed it, but they don't actually use it and wouldn't want it if
they had a choice about installing it.
Sachin Garg
"Dirk Stoecker" <dsto...@gmx.de> wrote in message
news:Pine.LNX.4.56.03...@kgins2.geo.tu-dresden.de...
> On Tue, 12 Aug 2003, Marco Schmidt wrote:
> For example all the bigger linux distributors (e.g. SuSE, Debian and
> RedHat) would very likely include an unsit tool in their distributions
> (like they did with the cabextract tool written by Stuart Caie), so
> the Linux market soon gets a nearly 100% coverage for decompression.
yeah!!! something like this may help.
> Alladdin already improves their compression technology over the time, so
> they will always will be in front of their competitors, as these could
> only reverse engineer and rebuild. I don't think that releasing the
> decompression would speed up the work on the side of competitors
> (Especially as the difference in complexity between compression and
> decompression algorithms is usually really large).
It may be true for LZ but not for PPM etc... as the model has to be same for
both compressor/decompressor.
Moreover, if there are x number of cool tricks you are playing in your
compressors, at least y number of them will be disclosed in the
decompressor. So, it *will* speed up the competitors, and thats the last
thing you would like to do in business.
Sachin Garg [India]
http://sachingarg.cjb.net
http://sachingarg.go.to
Dirk Gently
> > >use ZIP or RAR over StuffIt, in the same way they would rather use Windows
>
> You are right... I would rather use ZIP. That guarantees me the
> ability to open up that file 10 years from now.
>
> as a way of collecting files into one package. Sort of the "TAR" of
> the PC. The compression is just a nice bonus and rarely significant.
Me too. Although I'm not so sure that compression quality is
irrelevant.
Too bad people don't adopt the 7zip compression for the new zip. And
maybe bzip2. Those are both open source, which is what users need.
That's why programs such as ARC, ZOO, etc. disappeared. People
prefered the open format of ZIP.
> I suspect they've never actually optimized it for the x86
> architecture. And certainly not specific P3/P4/Athlon optimizations.
Good point. They probably haven't.
> They probably used the same type of "optimizations" for it that they
> did when they benchmarked their latest Mac against the latest P4 and
> "proved" how much faster their mac was.
[laugh] Yeah, that was rather funny.
> > is not going to win over the Windows platform. Maybe they don't care
> > about it.
>
> They don't.
>
> But you can't entirely blame Apple.
>
> With only, what, 12-10 million mac users left world wide (Linux has
[frown] Are you sure about that number? I know there aren't many but
I would have expected at least a few more. Certainly less than 20m
though. (Figure, what, 4m macs made each year, and that people keep
their old computer for a max of 4 years...)
You might be right, though. I've read a few reports that showed
newbies do tend to prefer WinXP over the Mac. And since the Mac has
always catered to the "brain dead" crowd, that would explain their
declining numbers.
> (Linux has
> almost as many desktop users!)
Really? I did read a report recently that said Linux was almost as
easy to use as XP now.
> can't really afford to cater to PC users. They've got to concentrate
> what little money they have into catering to their paying customers.
> If they don't keep them happy, then Apple may not exist at all in 10
> years, and wouldn't exist except as a "legacy" company in 5 years.
That is a very good point.
That also explains why a couple of Mac compression companies have been
in here talking about their compression formats and trying to get PC
users to switch!
Those companies are looking at the real chance of going out of
business in just a couple years because their own market is
disappearing.
As long as they stay on slower, more expensive hardware, that Mac will
never gain market share. If they were smart, they'd migrate to the
Athlon64. They've done a cpu switch before, and there are fewer users
now to be inconvenienced than there was before.
> Probably something like how Real & FLASH is... A lot of people have it
Does *anybody* like flash??! [shudder]
Marco Schmidt
[...]
>Too bad people don't adopt the 7zip compression for the new zip. And
>maybe bzip2. Those are both open source, which is what users need.
>That's why programs such as ARC, ZOO, etc. disappeared. People
>prefered the open format of ZIP.
Whatever the reason was, I think it's harder these days to make people
switch. There are a lot more people using computers today, and the
portion of users that are not interested in trying new technologies is
also much higher.
[...]
>Does *anybody* like flash??! [shudder]
Anybody who wants to have a website but can't be bothered with
providing actual content. ;-) No, that's unfair. Even usability guru
Jakob Nielsen has changed his negative evaluation
<http://www.useit.com/alertbox/20001029.html> of Flash:
<http://www.nngroup.com/reports/flash/>.
Regards,
Marco
Mikael Lundqvist
> Stuart Caie:
>
> [...]
>
>
>>Windows users would rather
>>use ZIP or RAR over StuffIt, in the same way they would rather use Windows
>>Media Player than Quicktime Player -- even though Quicktime may be "better"
>>in some technical aspect like compression or quality.
>
>
> Quicktime fails on the Windows platform because the playback is utter
> crap. A couple of months ago I couldn't play the large Matrix Reloaded
> movie trailer (.mov, 1000 x something pixels) on a modern system
> without a dramatic frame drop, which made it unwatchable. The 640 x
> something version worked, but it took almost all the CPU. Still no
> full screen playback with the 640 version without frame drop.
>
4, 1.7GHz) and had no problems with it. Maybe you *perceive* Quicktime
as bad because it's from Apple?
http://www.apple.com/trailers/
Quicktime is just a shell for codecs but obviously more powerful than
any of the alternatives since it was chosen to be a part of the Mpeg 4
standard. Also, it's hard to beat the quality in Sorenson 3 video in
full screen mode, it's very crisp and clean.
Your problems could be because of a badly configured Windows.
But think about it, 1024x720 or even 640x400 is not exactly a small
amount of data to move many times a second. I've looked at both WMV and
DivX movies in full screen mode and they were a lot worse in my eyes.
> I could reproduce that behaviour on different modern Intel / Windows
> systems.
>
What other applications were running at the same time?
> It's great if those trailers play smoothly on Macs, but that way Apple
> is not going to win over the Windows platform. Maybe they don't care
> about it.
>
That's not the problem.
People being lazy and ignorant is.
Why download an other player when you have one on the HD already?
Never mind that the alternatives may be better.
The "holy" platform war is another explanation.
/Mikael
Marco Schmidt
>I tested the movie trailers at Apple on the XP machine at work (Pentium
>4, 1.7GHz) and had no problems with it. Maybe you *perceive* Quicktime
>as bad because it's from Apple?
No, I have nothing against Apple.
>Quicktime is just a shell for codecs but obviously more powerful than
>any of the alternatives since it was chosen to be a part of the Mpeg 4
>standard. Also, it's hard to beat the quality in Sorenson 3 video in
>full screen mode, it's very crisp and clean.
Yes, the image quality is great, probably (I don't know for sure)
better than DVD. And I know about Quicktime being a container format,
but given that playback software, codecs and trailers are all
distributed from one place, I perceive that all together as the
"Quicktime experience". Besides, almost everybody uses their latest
codecs to encode video these days.
>Your problems could be because of a badly configured Windows.
I thought it may be a bad graphics driver, or bad support in a
graphics driver for whatever Quicktime uses. But as I said, I tested
on several modern Windows systems, with different hardware, and I got
lousy performance during playback from all of them. CPU usage maxed
out, therefore frames dropped with the high-res version of the
trailer. Other Windows users confirmed that they had the same
experience on their machines.
>But think about it, 1024x720 or even 640x400 is not exactly a small
>amount of data to move many times a second. I've looked at both WMV and
>DivX movies in full screen mode and they were a lot worse in my eyes.
Playing large DivX video files smoothly seems to be no problem, and
CPU usage is nowhere near 100%. The problem of QT / Windows
particularly seems to be upscaling, which I would consider a job of
the graphics card. When I watch a 640 x 400 AVI video it doesn't make
much difference in terms of CPU usage whether I watch it in fullscreen
mode (1600 x 1200 pixels) or as a simple window at normal 1:1 size.
With QT, I can watch the video of that size normally, but if I change
anything at the scale (increase the window a little, or double it in
size - 200% -, or go to full screen), playback becomes unbearably
slow.
>> I could reproduce that behaviour on different modern Intel / Windows
>> systems.
>>
>What other applications were running at the same time?
None. I tried everything to get those trailers running smoothly. Only
worked with the 640 version of the clip at original size, with nothing
else running at the same time.
>That's not the problem.
>People being lazy and ignorant is.
>Why download an other player when you have one on the HD already?
>Never mind that the alternatives may be better.
I don't see how laziness or ignorance apply here. Whoever wants to see
those trailers has to get Quicktime. It's not like any other software
would play it, at least with the modern codecs. And the Windows
implementation of QT seems to be somewhat inferior to what Apple is
doing on their home operating system. Maybe understandable that they
prefer their OS, but why give their QT platform a bad reputation by
annoying Windows users with low performance? It's not like anyone
would switch to Mac OS because they have to use QT so badly.
>The "holy" platform war is another explanation.
I can only talk for myself, and I don't care much for any OS
controversy. I regularly have to use three or four different OSs,
admittedly Mac OS not being one of them. But that is not my choice.
This rant is just about a particular Windows port of Apple software
that I dislike.
Regards,
Marco
Joe
Mikael Lundqvist <mlqn...@telia.com> wrote in message
news:qU40b.20379$mU6....@newsb.telia.net...
I have to think it's because of RAM issues. ):
Also: The likely reason QT was chosen, was political...
Dirk Gently
> >Too bad people don't adopt the 7zip compression for the new zip. And
> >maybe bzip2. Those are both open source, which is what users need.
> >That's why programs such as ARC, ZOO, etc. disappeared. People
> >prefered the open format of ZIP.
>
> Whatever the reason was, I think it's harder these days to make people
> switch. There are a lot more people using computers today, and the
That's true. But, if you are going to add some new format or ability,
it needs to be open source and easily added by all the clone makers.
*THAT* is why zip became so popular to begin with. That's why it is
still popular. Not because it's better because it was open which
allowed so many other people to also create programs for it.
RAR etc. are still adding new formats, but they aren't open. And I
don't think it's gaining much in the way of market share.
PKZip could have done the same thing, and people would have been
willing to follow it.
Then, in 6 months or so, after PowerArchiver, WinZip, InfoZip, etc.
had added it, people would just be selecting "maximum" compression
without even thinking about the algorithm underneath.
> portion of users that are not interested in trying new technologies is
> also much higher.
If you do it right, most of them will never even notice they are
trying new tech. They'll just be selecting "maximum" compression
(possibly by default) without ever even knowing.
(The same could be said about any sort of security improvement etc.)
But doing that will require a format to be totally open so that other
archivers can also implement it easily and freely.
Closed source (compression or otherwise) only works when you have
complete control over somebody and what they do. That isn't true in
the compression world.
> [...]
>
> >Does *anybody* like flash??! [shudder]
>
> Anybody who wants to have a website but can't be bothered with
> providing actual content. ;-) No, that's unfair. Even usability guru
> Jakob Nielsen has changed his negative evaluation
Well.. *I* have never, *ever* encountered a single flash site that was
easy to use. And of course, under IE, you have to enable ActiveX
controls and only a security inept newbie would blindly do that.
(This is off topic, though.)
Mikael Lundqvist
> But doing that will require a format to be totally open so that other
> archivers can also implement it easily and freely.
>
> Closed source (compression or otherwise) only works when you have
> complete control over somebody and what they do. That isn't true in
> the compression world.
>
source code but Joe and Jane Smith don't even know what source code is.
They do know though what "free" is.
They don't know how to use Winzip or just barely, and they have no clue
about any alternatives. They are happy as long as they can unpack the
archives they downloaded, containing a game of course because it's what
they mostly do with their computer, play games.
Pkzip or Winzip came free with the computer.
Joe and Jane are no rocket-scientists, don't treat them as such.
> Well.. *I* have never, *ever* encountered a single flash site that was
> easy to use. And of course, under IE, you have to enable ActiveX
> controls and only a security inept newbie would blindly do that.
> (This is off topic, though.)
:-)
--
Mikael Lundqvist
http://www.geocities.com/mikaellq/
Occam's Razor:
"Keep things simple!"
Kevin Easton
> Mikael Lundqvist:
>
>>I tested the movie trailers at Apple on the XP machine at work (Pentium
>>4, 1.7GHz) and had no problems with it. Maybe you *perceive* Quicktime
>>as bad because it's from Apple?
>
> No, I have nothing against Apple.
>
>>Quicktime is just a shell for codecs but obviously more powerful than
>>any of the alternatives since it was chosen to be a part of the Mpeg 4
>>standard. Also, it's hard to beat the quality in Sorenson 3 video in
>>full screen mode, it's very crisp and clean.
>
> Yes, the image quality is great, probably (I don't know for sure)
> better than DVD. And I know about Quicktime being a container format,
> but given that playback software, codecs and trailers are all
> distributed from one place, I perceive that all together as the
> "Quicktime experience". Besides, almost everybody uses their latest
> codecs to encode video these days.
>
>>Your problems could be because of a badly configured Windows.
>
> I thought it may be a bad graphics driver, or bad support in a
> graphics driver for whatever Quicktime uses. But as I said, I tested
> on several modern Windows systems, with different hardware, and I got
> lousy performance during playback from all of them. CPU usage maxed
> out, therefore frames dropped with the high-res version of the
> trailer. Other Windows users confirmed that they had the same
> experience on their machines.
I too have found Quicktime to perform lousily on Windows.
- Kevin.