RE: AMF Container Format - zip or no zip?

[Hod Lipson]

You are right - streaming will allow receivers to begin processing before the entire file transfer is complete.

My experience is, however, that the parsing time (~ 6 seconds per 1M triangles in AMF) is orders of magnitude smaller than processing time (e.g. geometric slicing).

But the cost of not having compression is detrimental, in terms of usability. File size is a major and growing concern.

In short, leaving compression out of the standard would be a mistake in my opinion, as it will encourage producers to produce large uncompressed files, and will create ambiguity as to who is responsible for compressing them. Most users don't care - they just need small files.

Whether or not it is zip or libz or something else is a good question. We chose zip because it is accessible to most non-experts and is widely used (I would argue it is the most common compression scheme). Non-experts would need this only if they wish to directly create a file themselves, something that was easily possible with STL and contributed to is adoption. But we could debate this.

---hod


-----Original Message-----
From: st...@googlegroups.com [mailto:st...@googlegroups.com] On Behalf Of rei...@13thmonkey.org
Sent: Friday, January 18, 2013 5:10 PM
To: st...@googlegroups.com
Subject: Re: AMF Container Format

Dear Hod,

On Wed, Jan 16, 2013 at 04:31:06PM +0000, Hod Lipson wrote:
> Streamability is not an issue: AMF files can't be streamed anyway because
> they are non-linear - you need the last vertex information before you can
> print anything.

Sure, but you can read the vertices while they come in ;) you don't have to
have received the entire file before you can start parsing :-) Thats also
streaming!

> The reason for including compression in the format is that it would be
> important for producer to produce a compressed file directly, rather than
> produce a non-compressed file and then compress it later. Otherwise, the
> intermediate file could be Gigabytes.

I have to disagree on this. Sure compresson is needed in some cases and
producing compressed files directly might be needed to avoid very big
intermediate files, that we agree on. What i dont agree with is the explicit
requirement/dictation to use ZIP with all its multi-file functionality as a
compression format.

When using libz for example, one can open a compressed file directly just as a
normal file, even on streaming stdout and just write the uncompressed data on
it and it will all be compressed on-the-fly and send out/written out. Its
compression is at par or better than zip. Its a single stream compresser.

Not dictating the compression format also leaves the interfacing parties to
communicate the details. I don't think a compression format should be part of
a standard since compression formats come and go. What might be hot and new
today is old and obsolete later.

Regardless of the chosen format, libz is free software and is free available
if not even standard on all unix systems while zip is very windows specific
but more importantly it used to be, and probably still is, propriatary; sure
there are readers and writers around but its hardly used outside windows
systems.

With regards,
Reinoud

--
You received this message because you are subscribed to the Google Groups "STL 2.0" group.
To post to this group, send email to st...@googlegroups.com.
To unsubscribe from this group, send email to stl2+uns...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/stl2?hl=en.

[Jesse McGatha]

On Saturday, January 19, 2013 1:53:20 PM UTC-8, Hod Lipson wrote:

You are right - streaming will allow receivers to begin processing before the entire file transfer is complete.

My experience is, however, that the parsing time (~ 6 seconds per 1M triangles in AMF) is orders of magnitude smaller than processing time (e.g. geometric slicing).

 

My main concern is with providing some common sense guidelines for folks who do want to optimize content for streaming (e.g. sorting triangles in Z order, with "bottom" triangles (those containing a vertex with the smallest Z value) first in the file, placing textures and materials definitions before they are referenced, etc. While I agree slicing and eventually printing take much longer than transfer, in a pipeline processing architecture, the transfer and buffering requirements could linearly scale with the number of processing components in the pipeline.

 

But the cost of not having compression is detrimental, in terms of usability. File size is a major and growing concern.

In short, leaving compression out of the standard would be a mistake in my opinion, as it will encourage producers to produce large uncompressed files, and will create ambiguity as to who is responsible for compressing them. Most users don't care - they just need small files.

 

I agree 100% with this. Not having compression intrinsic to the format would be a major barrier to adoption in the real world.

 

Whether or not it is zip or libz or something else is a good question. We chose zip because it is accessible to most non-experts and is widely used (I would argue it is the most common compression scheme). Non-experts would need this only if they wish to directly create a file themselves, something that was easily possible with STL and contributed to is adoption. But we could debate this.

 

ZIP is the only real compression choice for cross-platform, in my opinion. Other formats are not automatically available on Windows, which represents 90% of the market. Other formats are available on Windows, but not readily available on other platforms. 

 

 -Jesse

 

[Markus Hitter]

Am 19.01.2013 um 23:32 schrieb Jesse McGatha:

> ZIP is the only real compression choice for cross-platform, in my
> opinion.
> Other formats are not automatically available on Windows

If you enforce compression in the standard, the AMF producing tool
has to provide the file compression, so it's always available,
independently of what the operating system provides. Same for the AMF
reading tool.

For my part, I see the discussion on which compression algorithm to
use as a strong hint it shouldn't be part of the standard. AMF is
about describing 3D parts, not about concerns regarding file system
usage, streaming bandwidth or file system backup strategies.

If AMF were about file size, using XML would be an extremely poor
strategy to start with. Binary file formats are much smaller and -
for digital tools - much faster to read.


Markus Hitter

- - - - - - - - - - - - - - - - - - -
Dipl. Ing. (FH) Markus Hitter
http://www.jump-ing.de/

[Markus Hitter]

Am 19.01.2013 um 22:53 schrieb Hod Lipson:

> In short, leaving compression out of the standard would be a
> mistake in my opinion, as it will encourage producers to produce
> large uncompressed files, and will create ambiguity as to who is
> responsible for compressing them. Most users don't care - they just
> need small files.

IMHO, it doesn't exactly encourage them. It allows them. No more, no
less.

For my part I can't avoid to see an irony in this discussion. First,
XML is choosen, which is well known to be very verbose and well known
to produce very large files. Then, after having the format bolted
down, apparently ths opposite of the previous goals shall be fitted
on top of that.

So what we'll have to process then, is:

1. generate the geometry expressed in numbers

2. convert these numbers into a human readable format

3. fit these human readable numbers into a human readable XML scheme

4. compress this using some general purpose algorithm as a
countermeasure to the drawbacks of 2. and 3., also reverting the
advantages of 2. and 3.

- - file storage - file transfer - whatever - -

5. revert 4.

6. revert 3.

7. revert 2.

8. do something useful.

As every software developer knows, the results of 1. can be stored
into a file easily. And it'll likely be of a size similar to the
result of 4. Accordingly, steps 2., 3., 4., 5., 6. and 7. are just a
waste of processing resources.

If file size is suddenly a concern, please consider a binary format.
For example, a format equaling the current one, but expressed in C
headers instead of an XML schematic. Examples of successful binary
formats: JPEG, MPEG, STL.

[Hod Lipson]

Hi Markus,

I think that Compressed XML is a good compromise between the extensibility, readability, and popularity of XML, and the need for small file size.
Compressed XML also decouples these two conflicting design goals.

For reference, compressed AMF is approximately 50% smaller than compressed binary STL for the same geometry.

We might have been able to achieve slightly better compression using a non-standard binary format, but it would be more difficult to extend and understand. Note also that other mainstream formats (like Microsoft Word) also use compressed XML.

Given the various trade-offs that need to be met (of which file size and extensibility are just two) I believe this is a good choice.

--hod


-----Original Message-----
From: st...@googlegroups.com [mailto:st...@googlegroups.com] On Behalf Of Markus Hitter
Sent: Sunday, January 20, 2013 7:04 AM
To: st...@googlegroups.com

[Jesse McGatha]

On Sunday, January 20, 2013 3:54:21 AM UTC-8, Markus Hitter wrote:

If you enforce compression in the standard, the AMF producing tool  
has to provide the file compression, so it's always available,  
independently of what the operating system provides. Same for the AMF  
reading tool.

 

I understood that the human readability requirement in 3.1.2 indicated a preference for an ASCII format over binary formats (which are hard to understand and debug) and further stated that there was a way to read the file in an ASCII text viewer. This means that, regardless of compression implementations that producers or consumers may include in their implementation, there must also be a GUI way for an end user to decompress the file with a tool that is native to their chosen platform, thus making it viewable in a text editor native to that platform. ZIP is nice in that it has such universal support in the operating systems representing nearly the entire market share.

 

-Jesse McGatha

[Jacob Barhak]

Hi Hod,

You really should consider detaching the compression from the text standard. It really should be a wrapper. 

I am a great supporter of compression. I use it myself and typically use zip. Yet even zip has cross platform problems and behaved differently with different implementations. 

Also consider a user that gets a file named MyModel.AMF.ZIP. Should this user decompress the file once or twice to read the documentation in the text. 

In these small things you should really rely on existing conventions rather than force compression into the standard itself. Moreover with today's world where everything streams at such high rates, compression seems the least of your problems. It seems more of a nice to have feature that provides many advantages that are very useful. Perhaps add compression recommendations as an appendix to the standard?

Also, it seems people in the group quote existing tool preferences and market shares to support arguments. Note that this technology changes quickly. Consider the browser market. A few years ago most browsers where IE. Now it has all changed. 

If you want a standard to be robust and persistent through technology changes you should stick to the basics and avoid niches and dependence on existing market shares. Who knows in no time people will be using AMF from their tablets that may rely on Ubuntu/Windows/Android/OSX or any other device or even work on a virtual machine on the cloud. If a standard is developed with tools used today you may loose its generality in a short time. It's ok to look at what is dominating out there today, yet keep your mind open and try to keep things simple. If you succeed, tell me how. 

I hope this makes sense. 

          Jacob 

Sent from my iPhone

--

You received this message because you are subscribed to the Google Groups "STL 2.0" group.

To view this discussion on the web visit https://groups.google.com/d/msg/stl2/-/1t2oyPXzl1YJ.

[Jesse McGatha]

On Sunday, January 20, 2013 9:14:44 PM UTC-8, Jacob Barhak wrote:

You really should consider detaching the compression from the text standard. It really should be a wrapper. 

 

I truly believe that this is the wrong approach to take. For background, I worked for many years on the Windows print infrastructure, so my opinions are based on my experiences in that context, where file formats and compression really mattered to customers, so we received voluminous feedback from software developers, printer manufacturers, and enterprise IT departments. I believe 3D printing will be subject, long-term, to the exact same constraints that 2D printing has had to deal with.

 

While academically, it might seem that the right thing to do is to separate compression from being an intrinsic part of the file format, in real-world practice, the opposite is true. The feedback we consistently received, and finally re-worked the entire print infrastructure to accommodate, included the following sources of complaint:

 

1) On disk file size. Many enterprises mirror employees' data onto a backup share on their network. Many enterprises post files in collaborative file editing like SharePoint or source control systems, where version control duplicates all or a portion of these files. Some enterprises do not have personal storage options at all: all data is stored on the network storage facilities. As the size of data gets bigger, the direct cost of storing that data goes up. Files are emailed to other users and stored on shares, which consume more network storage and email server space. Print servers must spool jobs from all users until they can be sent to the target device; given the long 3D print times, this introduces a tremendous backup of files to print in the print queue, which must be stored on the centralized print server's disk for robustness/reliability sake. Enterprises want that data to be as small as possible in its on-disk format.

 

2) Network traffic. Related to the previous point, larger files create greater network traffic, resulting in overall degradation of network performance. This is further exacerbated by the centralized print queues most enterprises use, driven by a centralized set of print servers. Enterprises want the operating system and applications to conserve network traffic by having compact data representations.

 

3) Bandwidth cost. Further to the network traffic point above, enterprises with a WAN infrastructure between branch offices or those that have a large mobile workforce, incur an ongoing cost in bandwidth for sending data. Enterprises want this cost to be reduced as much as possible. In the case of 3D printing, where centralized print labs may be the norm for some time, this is particularly true.

 

4) Archival cost. Legislation often requires long-term archival of printed data. As AMF is intended to be an input format for printers, this will apply to AMF as well. This legislation requires storage of printed data for potentially decades. The ever-growing number of print jobs demands that the file format be as small as possible. Note that such legislation is often found in the medical and government contexts, principal users of 3D printing.

 

5) Supportability. Binary formats are hard for IT pros to diagnose and repair, which pushed toward human-readable formats such as XML. XML is inherently larger than binary, so compression becomes necessary to counter-balance this drawback.

 

6) Speed of transmission. It is not unusual to require files to be rapidly backed up, copied to alternate media, emailed, submitted to a printer, etc. To facilitate this, the smaller the transmission package the better.

 

I'm sure there were other considerations that I'm not thinking of off the top of my head as well. However, this shouldn't diminish the main point I'm trying to communicate: that real world customers actively and vociferously requested compressed file formats. They kept pushing on this until we were able to accommodate their demands. I truly believe it is the right thing to do and it is worth the effort to work through the challenges that it creates to include compression as an intrinsic part of the file format. To avoid the ambiguity between uncompressed and compressed versions of the format, I would far rather see them made explicitly different (different extensions and MIME types), or even better: do not support an uncompressed format for any purpose of interchange, much like .docx or .odf formats have no notion of an uncompressed format (although you can un-zip them into multiple files for human readability, editing, and debugging if you desire).

 

Jacob Barhak wrote:

 

Also, it seems people in the group quote existing tool preferences and market shares to support arguments. Note that this technology changes quickly. Consider the browser market. A few years ago most browsers where IE. Now it has all changed. 

  

 

Compression formats are not tied to any specific technology, but ubiquity in current market share is certainly a reasonable means to choose an easily adoptable compression format. Likewise, future platforms will have to consider the prevalence of certain file format compression mechanisms in choosing whether to provide support in their operating system. The key thing here is to define the compression format (ZIP, ideally with large file support) and the compression scheme (I'd strongly recommend DEFLATE), as well as address sufficient format-specific details (supported central directory records and what item headers are required) to allow an implementer to select an appropriate library to rely on.

 

-Jesse McGatha

[Jonathan Hiller]

I certainly would align with Jesse's slant on this. It may be
semantic, but the current amf compression IS a modular wrapper. By
defining the method in the standard we simply ensure that anyone who
wants to compress an amf file does it in a way that's globally
compatible.

Some quick googling reveals popular and well-supported apps for
decoding zip files on android and iOS (and all major desktop os's of
course), so I don't think this is a major practical concern.

I'm personally not opposed to having different file extensions (i.e
*.amf and *.amfz), but I haven't heard a convincing reason yet. (a) It
may confuse end users and non-experts who now wonder why there are two
types of files, (b) it's essentially no difference to those writing
software generating amf files, and (c) it's one additional line of
code for those writing parsers, and that line of code is defined in
the standard. In my mind (a) slightly outweighs (b) and (c). Thoughts?

~Jon

> --
> You received this message because you are subscribed to the Google Groups
> "STL 2.0" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/stl2/-/kELp3bfWCTAJ.

[Jacob Barhak]

Hi Jesse,

Your point about the usefulness of compression is accepted. We actually agree there. 

What I am trying to argue is that this is not a priority and you should use common sense when writing it in. 

Think about use cases: 

A user getting a file called: MyModel.AMF and wants to read the comments. 

Can the user open the file in a text editor? An XML editor? A specific application?

Does the user need to use 7zip to open the file? If so, should the user change the operating system definitions or the 7zip preferences to associate AMF with 7zip?

Consider another use case where the user/machine gets a file called MyModel.AMF.ZIP. How many times this file needs decompression to read the text? Is this number or unzips always the same for different files with such an extension?

This is not a clear cut definition. 

To avoid these ambiguities you can choose the following routes:

1. All AMF files are compressed using zip - cannot be opened using a text editor. 

2. All AMF files are either compressed or not compressed. The extension implies that only an AMF application or an expert know how to open the file and figure out details. 

3. All AMF files are text files not compressed and the user can choose if these should be compressed to add a ZIP extension after AMF.

I suggest to go with the third option since it has several advantages:

1. The standard is unambiguous while still allowing users to view the file and choose compression. 

2. It can be implemented using exiting off the shelf components for compression that allows pipelining AMF and ZIP. 

3. It simplifies the AMF encoder/decoder software and allows it to focus on what it does best - deal with geometry - not compression or other things. 

4. The standard can still mention compression in an appendix, while stating that it will add a compression extension to the file name that will otherwise end with AMF. This way ambiguity is avoided while allowing the users and implementors to understand what is going on. To my preferences the appendix should be optional for implementation yet highly recommended. The AMF should be a text format that people can look inside rather than a proprietary format that users only see as a black box and only readable by experts. 

5. It allows future extensions to the standard to cope with transmission of the information and possible other compression methods. If the standard becomes widely used you will really need those extensions for some applications. 

I hope I was able to properly convey and justify my thoughts. 

        Jacob 

Sent from my iPhone

--

You received this message because you are subscribed to the Google Groups "STL 2.0" group.

To view this discussion on the web visit https://groups.google.com/d/msg/stl2/-/kELp3bfWCTAJ.

[Hod Lipson]

I think compression is critical – if CAD systems will produce uncompressed AMF in any but the simplest files, the format will be impractical, no matter how good the rest of it is. And if its going to be compressed, we should specify exactly how so that there is no ambiguity, and I would go with something popular and stable, like zip.

 

BTW STL also had an ascii and binary version with the same extension, and most readers and users had no problem dealing with both.

 

--hod

 

From: st...@googlegroups.com [mailto:st...@googlegroups.com] On Behalf Of Jacob Barhak

Sent: Monday, January 21, 2013 12:57 PM
To: st...@googlegroups.com

Subject: Re: AMF Container Format - zip or no zip?

 

Hi Jesse,

[Jesse McGatha]

 

I actually question the usefulness of an uncompressed format (for interchange). Who actually needs the data uncompressed? As near as I can tell, it is only experts that might want to work with uncompressed data using OS tools while building systems or hand editing models. These experts could easily know that a ".amf" extension on the file means "ZIP archive containing an XML ZIP item with AMF data in it". The expert could then unzip the file should they want to view it in a text editor (that is, only experts ever want to look at the file in a text editor) and re-ZIP it when they want to share with another user or tool for interoperability. It simplifies implementers' lives if they only ever have to deal with one (compressed) format and keeps the ecosystem clean by rejecting uncompressed files. The overwhelming recommendation of the spec is that implementers should only use the compressed format, given the evidence in the performance annex. The real world scenarios urge usage of only the compressed format. I think the spec should acknowledge that groundswell and just state that the format is always compressed. This simplifies everything.

 

-Jesse McGatha

[Martin Wicke]

I agree with this. It's true that there's effectively no cost to
implementing the check for 'PK', but even just allowing for uncompressed
AMF will lead to actual uncompressed AMFs out there. I don't think
anybody wants that.

I also second the notion that no human will ever look at AMFs
(hopefully, otherwise, something went wrong). They are human readable
only in the most trivial meaning of the word. ASCII STLs, by the way,
are human readable too. How many people have actually ever edited an STL
file by hand? I've looked at them to ascertain that they're not
truncated, but editing vertices etc? Never.

Maybe materials etc make that more of a possibility, but that would just
mean that the programs out there that can be used for viewing and
editing AMF files are doing a particularly crappy job. I don't think
it's worth considering that case. If users resort to text editors over
proper tools for access to AMF features, AMF has failed.

The only exception I would allow for that is developers of AMF producers
or consumers. But they, having read the standard, know about the secret
with the zip, so they'll have no trouble either way.

Martin

> --
> You received this message because you are subscribed to the Google
> Groups "STL 2.0" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/stl2/-/NjZHtuFwZ7AJ.

[Jonathan Hiller]

The main reason to support uncompressed amf (besides debugging
purposes) is to support the hacker/grad student/enthusiast directly
fprint()ing amf files from their simple matlab/python/custom scripts.
It's not a deal-breaker since we'll publish the zip "secret", but to
the inexperienced programmer (or one working in a programming
environment without good zip support) adding a necessary compression
step is adding a barrier to adoption at the grassroots level.

I most certainly have ascii stl files on my hard drive that I created
for quick hacking/testing. I think the analogous ascii amf is helpful.

~Jon

[BobC]

This is an interesting one, I have been unsure which side of the fence to be on. I think that in principle compactness should be a fundamental part of the spec (like in JPEG), or not at all (like in HTML).

I can see the practical benefits of compact format for storage and transmission, but really these are transport layer issues, not application ones. Even so, users currently zip/rar all manner of files for storage and transmission, I think they are quite able to do that themselves.

If I was going for a compact/compressed format, I would look to a universal scheme like XMill rather than yet another application specific implementation. It is interesting that binary XML formats created interest 10 years ago, but have not developed since. Does this imply that advances in capacity mean it is no longer required?

On Monday, January 21, 2013 8:59:23 PM UTC, Martin wrote:

I agree with this. It's true that there's effectively no cost to
implementing the check for 'PK', but even just allowing for uncompressed
AMF will lead to actual uncompressed AMFs out there. I don't think
anybody wants that.

I think this is persuasive for me. Most STLs I see are text, and I have had problems with applications not supporting binary ones properly. History shows if you allow multiple formats, it creates more compatibility issues. It really would be better to have a single format in the standard, whether compressed or not.
 
If developers need to look at the uncompressed data, it is trivial for them to do that, it is not necessary for it to be part of the standard.

Lastly, I would note that I haven't actually seen any packages supporting AMF. I know it is tempting to fix up AMF with lots of perceived improvements, but perhaps we should let it establish itself as a viable format with widespread support before creating a new version? We don't want to end up like VRML 2.0.

[Jacob Barhak]

Hi Hod,

It seems you chose the option of AMF meaning zip. 

It is an ok choice to avoid an ambiguity. It implies the following:

The standard user sees a black box while an expert can access the data and documentation. 

It also means that file size is generally small. Lets say a magnitude smaller for the sake of argument. 

From now on I will play the devils advocate so you can either change your mind, or reaffirm your choice. 

First. Let us examine if compression is critical or just desirable. 

What is the largest file size you want to represent with AMF ever?  How many triangles? How many GB?

You can try and find the biggest example you have access to now and multiply it by 10.

How much time does it take to copy such a file via USB 3.0? How much using a fast network?

Zipping the file may reduce this time by a magnitude and allow it to be ported into portable memory.  

This is an advantage. How much time does this take? Now you have a quantifiable idea on what you gain by putting it in the standard and you have an idea of limitations and tradeoffs. 

Now think about cloud environment. Does it really matter in this environment? Think about a thin client printing machine with only a buffer, network card and a controller. It gets the file after already sliced in the cloud service that just communicates the next slice to the printer. The printer is much cheaper and the software in the cloud is less prone to size issues. Such a design may be more economic. And under such a design the CPU time needed to zip/unzip may play against you under some criteria. Yet if you decouple ZIP and AMF into 2 stages where ZIP encapsulates AMF only upon demand you may play on both fields: cloud and dedicated printer and be much more scalable. 

Moreover think about a single AMF file processed in parallel and even printed in parallel using multiple steams. How do you slice a zip file? Yet you may be able to intelligently slice a text file. 

Your lab specializes on printing machines. You should create such a cloud design and release it either freely or as perhaps patent it. 

And here is another example where compression may play against you.

When you add the word zip in the standard you must be very specific with the link since implementations are different. Is there a specific standard that defines zip you can point to with a specific version? 

What I am trying to say is that once you add dependency on zip in your standard you have to support it as part of your own standard. After all the user gets an AMF file, not a zip. You are now responsible. 

Just the other week I transferred a zip file from an Ubuntu machine to a windows 7 machine. The windows 7 system could not properly uncompress some files in the archive. Yet 7zip was able to extract the same files from the same archive on the same machine. I still have the file to demonstrate this. This means that zip implementations differ, and although rare, you can find compatibility problems. Now imagine that this happens to a user that sees an AMF extension. This user now thinks the AMF standard creates corrupt files. The user does not know the problem originated with zip. 

The reason I supported the option of AMF.ZIP to indicate compression is since it actually avoids the above problems to some extent. 

You still have to be exact in identifying the compression format version. So you are still dependent, yet in case of a problem the non expert user may be able to solve it on their own. And in some cases the user/machine may choose to avoid compression. 

You can always state in the standard that by default a file will be compressed and will have the .amf.zip extension. And this default can be overridden by the user to generate an uncompressed .amf file. This way you get the best of all worlds while still keeping your preference by hard coding a default compression behavior in the standard that implementors have to conform to. 

There are other advantages of being open to the user such as allowing the user to view documentation within the file as text without a special application. By the way, do you allow Unicode characters for documentation to support multiple languages?

If after this devilish advocacy you decide to go a head with your decision, then you have done this after considering possible consequences and trying to look into possible future use cases. 

By the way, your decision is good as long as ambiguity is removed. I am just trying to see if there is an optimal solution that will accommodate more options and better survive changes in technology. Yet predicting the future is hard, so I may be on the wrong track. 

I hope it was worth your time to read this. 

             Jacob

Sent from my iPhone

--

You received this message because you are subscribed to the Google Groups "STL 2.0" group.

To view this discussion on the web visit https://groups.google.com/d/msg/stl2/-/2bQi4GB5Vs8J.

[Reinoud Zandijk]

Hi folks,

I agree with you Jacob on your analysis of AMF use. I do fullhartly support
having a compression suffix to compressed AMF files. This avoids ambiguity.

As for Zip compatibility, yes, i've been bitten by this for quite some time.
Also the standard conforming zip has a file limit of 2 Gb! There are
extensions of the format to cater for it, but as you might guess, it *can* and
so *does* get problematic. Also the zip standard extensions don't allways mean
that they can interchange correctly.

As for streaming, someone on this thread told me that a ZIP file can be
tweaked to have its header in front, but that won't happen in the wild.
Streaming ZIPs thus also means that some programs might need to buffer quite a
lot until all info has come by for them to decode the stream; this can mean
multiple megabytes/gigabytes in odd cases.

You see, ZIP is not really a compression tool, its an archiver with buildin
compression.

Thats a huge difference.

If you want streaming, you dont want ZIP period. You might want zlib. Its the
library gzip(1) is just a shell around. It can be started to read as soon as
the 1st byte comes in where as ZIP needs its index to come by.

If compressed output is wanted directly, zlib also offers direct mapping on
fopen/fprintf/fgetc/fgets and friends calls; programs thus don't need to be
modified much to auto-include writing out compressed output. The compression
ratio can be specified as a level between 0 (just storage) and 9 (best
compression).

Using gzip, its main user, it will also allow things like this :

zcat file.amf.gz | amf_strip_tex | ./my_script | gzip > file-out.amf.gz


One note about compression though: I've found that compression and
decompression of huge files in the Gb range take heaps of processor power and
take quite a while. If every program in pipelines need to decompress and
compress its a waste of CPU power to say the least!

The reason OpenOffice and MS Word can use zip's with compressed XML's in them
is that they don't allow/want streaming but in stead update statically stored
files. That makes a huge difference.

One final note on compatibility, the proposed zlib is and has been available
for quite a while and even to MS Windows. Almost all, if not all archivers
i've seen under MS Windows recognize and decompress zlib's files too. It has
been used for quite a while alredy by ssh(1), cvs(1) and various other
streaming tools.

Rounding up, i'm all for uncompressed and compressed AMF files, with
compressed files having a compression suffix, maybe even `.amz' or `.amfz'.
Also the standard could say its recomended that authoring programs store their
files preferably in compressed form for storage issues. The compression format
shall feature direct streaming without buffering needs.

With regards,
Reinoud

[Hod Lipson]

 

Here are a few data points:

 

Compressed AMF is approximately 20 times smaller than uncompressed AMF .

Compressed AMF is about half the size of compressed binary STL

A typical file with 1 million triangles will be 200MB uncompressed AMF, or about 10MB compressed AMF.

   (This means that for most people, an uncompressed AMF cannot be emailed; a compressed file can.)

 

Bottom line, I think we should stick to zip compression for practical reasons, even though from an academic point of view I agree that compression is a separate issue. Compressed XML decouples these two design goals as much as possible.

 

I also agree with BobC that at this point we should wait for AMF adoption before improving it.

 

--hod

 

 

 

Hod Lipson

Associate Prof. of Mechanical & Aerospace Engineering and Computing & Information Science

Cornell University, 242 Upson Hall, Ithaca NY 14853, USA

Office: (607) 255 1686 Lab: (607) 254 8940 Fax: (607) 255 1222

Email: Hod.L...@cornell.edu

Web: http://www.mae.cornell.edu/lipson

Administrative Assistant:  Craig Ryan  cd...@cornell.edu, (607) 255-6981, Upson 258

Calendar: http://www.mae.cornell.edu/lipson/calendar.htm

 

From: st...@googlegroups.com [mailto:st...@googlegroups.com] On Behalf Of BobC
Sent: Monday, January 21, 2013 5:28 PM
To: st...@googlegroups.com
Subject: Re: AMF Container Format - zip or no zip?

 

This is an interesting one, I have been unsure which side of the fence to be on. I think that in principle compactness should be a fundamental part of the spec (like in JPEG), or not at all (like in HTML).

--

You received this message because you are subscribed to the Google Groups "STL 2.0" group.

To view this discussion on the web visit https://groups.google.com/d/msg/stl2/-/2bQi4GB5Vs8J.

[Markus Hitter]

Am 22.01.2013 um 18:52 schrieb Hod Lipson:

> Compressed AMF is about half the size of compressed binary STL

This is expected, as STL holds a lot of duplicate data. Binary AMF
might be even smaller.

> Bottom line, I think we should stick to zip compression for
> practical reasons, even though from an academic point of view I
> agree that compression is a separate issue.

I think about all people here agree compressing files with human
readable data before sending them somewhere is advisable. So far I
can't see much evidence zip is the best tool for this task, other
than its popularity among Windows users.

Side note: zip and gzip use the same compression algorithm, so sizes
are always comparable.

Advantages of zlib/gzip over zip I've seen in this discussion:

1/ streamable without special tweaks

2/ works for files of any size, also files > 4 GiB uncompressed

3/ no archive headers -> easier to implement

Advantages of external compression tools:

4/ uncompressed data is valid data, so it can be read and written by
quickly hacked tools

5/ tools with higher compression ratios than zip/gzip are allowed

6/ less text in the AMF standard -> even easier to implement, less
standard maintenance

Now I'm also well aware it isn't easy to change people's mind once
they settled for a choice. In case compression has to find it's way
into the AMF standard, I'd wish to have at least 1/, 2/ and 4/
adressed some way or another.


Markus

[Martin Wicke]

Markus Hitter wrote:

> Side note: zip and gzip use the same compression algorithm, so sizes are
> always comparable.

zip can, but doesn't have to, use the same compression as gzip. It may
be a good idea to specify this in the standard to avoid trouble. zip
allows for different compression types that may or may not be supported
by different zip/unzip implementations.

According to http://www.pkware.com/documents/casestudies/APPNOTE.TXT,
zip allows for 17 (or maybe 13, depending on how you count) different
compression algorithms, the majority of which I doubt are implemented in
the tools shipping with Windows/Mac/whatever. MacOS for example supports
only deflate and bzip2.

This has come up before, it's not all that easy to point to an
authoritative zip spec (the file linked above may be it), and zip is
extremely broad.

If we do specify some more than just "use zip", this may well be fine.
At the very least, we should probably say "use zip, compression deflate,
no encryption, compatible with ZIP version 4.5 (the minimum needed to
use ZIP64, which is necessary for more than 2GB files).

> 1/ streamable without special tweaks

I'm not all that convinced by the streaming argument. Unless we add
strong additional requirements on how the geometry is stored, pretty
much the only thing you can do with the streamed file is parsing. This
may still be worth it for files whose XML doesn't fit in memory but
their data does, but I don't think that's much of a concern (since files
that big will very almost certainly blow up the printer software, and
that won't change -- as the printers get better, so does memory size).
Almost all operations on geometry (slicing, validation, etc) can only be
done once the geometry is complete (again, unless we enforce additional
constraints on the order of primitives).

> 2/ works for files of any size, also files > 4 GiB uncompressed

ZIP64 should take care of this -- is it universally supported? Probably
not (mimiz.c does not), but at least the MacOS/Linux versions of zip
support it.

> 3/ no archive headers -> easier to implement

Definitely. I love gzstream myself. Completely transparent. Only a
single file though. That's the tradeoff.

> 4/ uncompressed data is valid data, so it can be read and written by
> quickly hacked tools

Not sure that's much of an argument, virtually all scripting languages
support zip well: there's a zipfile module for python, Archive::Zip for
perl, a zip() function in matlab, and zip/unzip command line utilities
on both Linux and MacOS. It's one extra step, yes, but really not that
much of a drag (use system("zip -j <path_to_compressed_amf>
<path_to_uncompressed_amf>") in C/C++ ).

> 5/ tools with higher compression ratios than zip/gzip are allowed

bzip2 is allowed in zip starting at version 4.6 (we could require that
and allow bzip2 compression also if that's an issue). However, I doubt
that the difference between deflate and bz2 will ever make a difference.

> 6/ less text in the AMF standard -> even easier to implement, less
> standard maintenance

Arguably, since Zip 4.5 (or 4.6), without encryption and using deflate
(or possibly bzip2) will never change again, this is not an issue.

I think the main issue is that we should really say exactly what we mean
by "zip". I think the other concerns are no major problems. I will allow
for the possibility that I totally fail to see some obvious advantage in
streaming, which zip definitely does not support (ie., which maybe some
implementation somewhere supports, but which is not near widely enough
available).

All that said, I do think that just deflating the stream would be a lot
easier for the developers. But that's really the only argument. The
downside I can see with that compared to zip is that it's one file only.
Bad for manifest or eventual external data (textures etc). And given
that choice, zip seems like a winner to me.

Martin

[Reinoud Zandijk]

On Tue, Jan 22, 2013 at 05:33:25PM -0800, Martin Wicke wrote:
> >5/ tools with higher compression ratios than zip/gzip are allowed
> bzip2 is allowed in zip starting at version 4.6 (we could require that
> and allow bzip2 compression also if that's an issue). However, I doubt
> that the difference between deflate and bz2 will ever make a difference.

why not use libarchive then? :) though yes, its not as transparant as zlib...

> >6/ less text in the AMF standard -> even easier to implement, less
> >standard maintenance
>

> All that said, I do think that just deflating the stream would be a lot
> easier for the developers. But that's really the only argument. The
> downside I can see with that compared to zip is that it's one file only.
> Bad for manifest or eventual external data (textures etc). And given
> that choice, zip seems like a winner to me.

A manifest is only needed when there are more than one file and doesn't really
make much sense since there are metadata tags defined in AMF already; they
only need a form of standardisation as in: who designed it, what company, when
it was created, last update etc.etc. If its in the stream is far more likely
to be correct than a separate file.

Apart from being easier for developers i can name some more benefits of just
deflating the XML file: streaming and consistency!!

Yes, streaming might be important in printer's applications since they may
want to pre-process the files before printing. Just as for normal 2d printers
this is most likely normally done by runing filters. It might f.e. filter out
the textures, it might want to adjust textures for their specific printer
(sharpness, brightness, colorspace), it might want to strip all data that
their printer gets confused about or is not relevant to the printer job or to
enforce a specific AMF version, help prevent triggering bugs in the printers
AMF support etc.etc. If each of these operations need to unzip, read the
entire file, process it, write it out and zip it again its a waste of
resources and most of all time.

Also, when all is in one file, the internal consistency is more guaranteed
esp. with images. A `clever' user might want to update say a texture in the
zip by f.e. changing a file format or changing the image resolution. Stuff
might still look fine on intermediate steps but eventually breaking stuff at
the printer's end since say the printer doesn't understand subformat X of say
a jpeg or a gif/png. He/she may also delete big textures in the zip to make
it sendable by mail, breaking references etc.etc. One file, one either
valid or not.

With regards,
Reinoud

[Chester John]

Just a though, you could specify a well defined version of zip such as the java jar file format, and then use a manifest for specific details on file handling.


Chester John

[Jacob Barhak]

Hi Hod,

You are correct. This seems all academic. 

If the standard version is closed and the Wikipedia page is up to date then the solution was already decided and AMF can be either zipped or not. 

So if this discussion continues, people should take into account that this is a debate to generate ideas for next versions.

I have a question out of ignorance, who decides on wrapping a version for the standard and is there an agreed mechanism for that? 

Never the less, I must add that this discussion was interesting enough to draw me away from other matters. 

The group has good ideas. 

         Jacob

Sent from my iPhone

<ccjohn.vcf>

[Hod Lipson]

Hi Jacob,

 

The decision to change or extend the standard is carried out by voting by the members of the ASTM F42 “Additive Manufacturing” group.

Anyone can propose a change; and anyone can join the group and vote. Every company gets one vote regardless of how many representative members it has.

(see ASTM protocol for details).

 

In case you are wondering, I have volunteered to lead the discussion but I have no final say on what gets proposed or approved or not.

 

--hod

 

Hod Lipson

Associate Prof. of Mechanical & Aerospace Engineering and Computing & Information Science

Cornell University, 242 Upson Hall, Ithaca NY 14853, USA

Office: (607) 255 1686 Lab: (607) 254 8940 Fax: (607) 255 1222

Email: Hod.L...@cornell.edu

Web: http://www.mae.cornell.edu/lipson

Administrative Assistant:  Craig Ryan  cd...@cornell.edu, (607) 255-6981, Upson 258

Calendar: http://www.mae.cornell.edu/lipson/calendar.htm

 

From: st...@googlegroups.com [mailto:st...@googlegroups.com] On Behalf Of Jacob Barhak
Sent: Wednesday, January 23, 2013 5:04 PM
To: st...@googlegroups.com
Subject: Re: AMF Container Format - zip or no zip?

 

Hi Hod,

[Jacob Barhak]

Thanks Hod,

In that case, when is the next time changes will be made possible?

If some conclusions will be reached by the group before that date they may be raised to the committee. 

I saw that the Wikipedia page was last updated on 4-January. And it seems like updates are frequent at the monthly level. 

Does this standard evolve fast just like software libraries? Which version is prominent? I know that software libraries sometimes take years to be fully adopted many times due to dependencies with other libraries. 

You had a good point about practicality in a previous post. Sometimes it is worthwhile to cut the Gordian knot. The rate of evolution should also be considered when raising all those ideas. As well as backward compatibility. 

Yet I will congratulate again the good brain storming going on here. 

       Jacob

Sent from my iPhone

Visit this group at http://groups.google.com/group/stl2?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Hod Lipson]

 

I believe that we should generate a new revision soon to (a) clarify curved triangles and (b) provide make the schema official. There are also various minor corrections and clarifications. I can send the word doc for anyone interested to edit (with “track changes” please).

 

As for the compression, I do not see any consensus for change at this point, so I suggest we leave as is for now.

 

Once the new revision converges, we post it on the ASTM website and members are notified. They then have a certain period (I think one month) to challenge the changes. Then voting happens. If even one vote is negative, the change doesn’t pass. So it is best to reach a consensus before voting.

[BobC]

On Wednesday, January 23, 2013 10:47:25 PM UTC, Hod Lipson wrote:

 

I believe that we should generate a new revision soon to (a) clarify curved triangles and (b) provide make the schema official. There are also various minor corrections and clarifications. I can send the word doc for anyone interested to edit (with “track changes” please).

 

As for the compression, I do not see any consensus for change at this point, so I suggest we leave as is for now.

I think the discussion has been useful, and that is a reasonable conclusion.

The way I currently interpret the spec (Section 12), is "an application may generate either a compressed or uncompressed AMF. An application must be able to read both compressed and uncompressed AMF". Does that sound correct?

Possibly it should be read as: "an application should generate compressed AMF and may generate uncompressed AMF"

(where "may","must" and "should" are as generally used in formal specs).

I also noticed when researching what other standards use ZIP format and how they deal with it, that ISO recommended that it is sufficient for standards that use ZIP compression to refer to APPNOTE.TXT source : http://en.wikipedia.org/wiki/Zip_%28file_format%29#Standardization

Which is what the AMF spec does, so it seems that was considered at the time of writing.