community guidence & testing please: AppleSingle signature development

[Andrea Byrne]

After being inspired by Jenny Mitcham’s blog post  about signature development and the encouragement of my colleague at Archives New Zealand, Ross Spencer, I decided to try my hand at creating a signature as well. I noticed there were a few folders of unidentified PDFs in the govdocs corpus, and, after some investigation, realized some were wrapped in a lightweight container format called AppleSingle that Apple developed to store Mac OS files across different filesystems (http://fileformats.archiveteam.org/wiki/AppleSingle).  I didn’t see a puid for AppleSingle in PRONOM, and decided to take a crack.

 

In the documentation (see http://macjournals.com/special/AppleSingle-Double.pdf  http://apple2online.com/web_documents/ft_e0.0001_applesingle.pdf  and https://tools.ietf.org/html/rfc1740)  and confirmed by the sample files, the AppleSingle version 2 format can be identified as such:

A 4 byte Magic Number that is always at the beginning of the file

00 05 16 00

Followed by a 4 byte version number

00 02 00 00.

(Version one’s version byte sequence is 00 01 00 00, but as I didn’t have any version 1 files at the ready, I didn’t have anything to test against. I’ll attach my sig for version 1 as well, but with that caveat)

Those 8 bytes are followed by a 16 byte filler. All my sample files had fillers that were all zeroes, but I saw some documentation that said that home file system data could be kept in this filler, so I left that part of the header out of the signature.

The ‘regular’ body of the format (ie PDF) follows the AppleSingle header, so AppleSingle acts like kind of a container, I suppose. I don’t know if PRONOM/Droid is open to introducing a new kind of container format, or what the history of PRONOM and these kinds of Apple containers is (after looking around, I found references to a few others that Apple created, AppleDouble and MacBinary that I want to look into as well). For preservation purposes, it should probably at least be noted in any future PRONOM entry that any files identified as this format deserve investigation into the payload, because it’s likely that AppleSingle contains another file type.

 

The signature I’ve created for Version 2 works for the AppleSingle files in the govdocs corpus. I also ran it against Ross Spencer’s skeleton suite, and the entire govdocs corpus and got no false positives.

 

As part of this thread I’m looking for:

•     Community testing of the signature

•     More sample files of AppleSingle (both version 1 and version 2) to test against
  

•       Guidance from the PRONOM team on how we should create entries in PRONOM, and if this should be a new kind of container format or not
  

I am attaching some of the sample files from govdocs, as well the signature for version 1 and version 2, again noting that I don’t have any version 1 AppleSingle files. It just seemed funny to write a sig for version 2 without their being a version 1!

 

Thanks for your help and consideration,

Andrea Byrne

[Dclipsham]

Thanks Andrea, this is awesome!

AppleSingle is proving rather elusive and I've yet to find any in our own collections so far, but I'll keep looking

I should note, we do have a full entry for AppleDouble: http://www.nationalarchives.gov.uk/PRONOM/fmt/503

And we have an outline entry (albeit with a signature) for 'BinHex Binary Text', (.hqx extension) - I wonder if this is the same as MacBinary? - http://www.nationalarchives.gov.uk/PRONOM/x-fmt/416 Either way this entry needs a bit of love..

More info on all three here: https://www.iana.org/assignments/media-types/application/applefile

The references you've provided are pretty clear so I'm more than happy to add the entry for AppleSingle, noting in the format description that investigation of the payload would be prudent. Were you able to find any reference to the differences between versions 1 and 2?

When submitting signatures, it's easiest for us if you tell us the proposed signature string(s), e.g. BOF: 0005160000020000 and the rationale behind the signature (in this case the spec is extremely clear). It's really nice for us if contributors flesh the entry out, for example providing MIME information (we only add if it's an 'official' MIME - either declared by the format vendor, or listed on the IANA Media Type list - http://www.iana.org/assignments/media-types/media-types.xhtml) and descriptive text (which should by objective - no 'format xxx is the fastest...' or '...uses the best encryption method').

I think any interrogation of the 'container' should be driven by the identification tool, so I could envision DROID, for example, probing the contents of AppleSingle further and identifying the underlying format (Do you know if the relationship is always 1:1? i.e. AppleSingle only wraps a single file, or whether it can aggregate many?). This would be similar to how DROID probes zips, gzips, w/arc.

The DROID container signature mechanism represents a slightly different scenario where the format is an aggregate of multiple fragments or files and the wrapper and contents as a whole represent a single intellectual entity - e.g. a Word 2007 .docx file is a wrapper containing a host of .xml files and any other entities (e.g. embedded images) that form the Word Document.

Hope this helps. I'll add AppleSingle in September's release. 

David

[Dclipsham]

Yeah, BinHex definitely isn't the same as MacBinary. Looking at the (almost entirely commented out) entry for MacBinary in the File magic library. MacBinary looks tricky to say the least, at least up to version 3  https://github.com/file/file/blob/master/magic/Magdir/macintosh

David

[Dclipsham]

I wrote:

 

The references you've provided are pretty clear so I'm more than happy to add the entry for AppleSingle, noting in the format description that investigation of the payload would be prudent. Were you able to find any reference to the differences between versions 1 and 2?

That'll teach me to actually read rather than just scan. Appendix on page 21 of http://macjournals.com/special/AppleSingle-Double.pdf covers at least some of the differences, so I'm very much of a mind to have separate entries for versions 1 and 2, and to add a version 1 entry for AppleDouble.

 

[Andrea Byrne]

 

Thanks for looking into this, David! And for your tips about submitting file formats.

 

For the extra information for the PRONOM entry:

 

The official mimetype is  listed  as application/applefile

 

Description:

Apple Computers created the AppleSingle format (as well as the AppleDouble format) to represent and preserve the attributes of files across files systems that do not support the same attributes of the file’s home systems. AppleSingle is primarily a storage format and contains both a file’s contents and attributes. It consists of a header and one or more optional data entries followed by the byte stream of the file it is storing. It maintains the original Macintosh filename and file type(ie. Mydoc.pdf). The AppleSingle format can store a number of different kinds of formats.

 

Specific to AppleSingle v2:  AppleSingle 2.0 is a revision to the original AppleSingle specification, in that it replaced and added some data entry fields

Release: 1990

 

Developed by: Apple Computers

 

Documentation: https://tools.ietf.org/html/rfc1740 http://apple2online.com/web_documents/ft_e0.0001_applesingle.pdf http://macjournals.com/special/AppleSingle-Double.pdf

 

Signature description:

BOF: “….....” 4 byte AppleSingle Magic Number, followed by 4 byte version number

 

I’m still doing some research on the format, and would be happy to flesh the entry out more if needed.

 

Andy Jackson was kind enough to search the UK Web Archive for files that include the AppleSingle magic number. Because of ANZ proxy stuff, I’ve been unable to download the entire payload so far, but I have a few more examples (that aren’t pdf!) I can share from that collection.  As far as I know from what I’ve read and looked at, the relationship is 1:1.

 

MacBinary does look tricky, and I was going to try and tackle that next. I have a few examples from the govdocs corpus, and the spec for macbinary II (http://files.stairways.com/other/macbinaryii-standard-info.txt)says:

 

It is possible to write a much more robust routine, by checking the

following:

  Offsets 101-125, Byte, should all be 0.

  Offset 2, Byte, (the length of the file name) should be in the range of 1-63.

  Offsets 83 and 87, Long Word, (the length of the forks) should be in the

    range of 0-$007F FFFF

 

So… fun! I’ll try and take a look into that 'BinHex Binary Text' format, too. These Apple header type files are a rabbit hole I’m happy to fall into.

Attachments:

•  CSV of files from the UK Web Archive that have the AppleSingle magic number
• A pch, mpe and bin file with the AppleSingle header
• Two PDFs with the macbinary header
  

 

 

[Andrea Byrne]

Also, here's a link to the results from the UK Web Archive query: https://www.webarchive.org.uk/shine/search?query=content_ffb%3A%2200051600%22&sort=crawl_date

[Dclipsham]

Great stuff! 

I found a version 1 in the web archive collection pretty quickly:

http://web.archive.org/web/19970428134800/http://nasc.nott.ac.uk:80/ftp/pub/dist/Vol3iii/leicester/weedsworld.dec96ms 

Successfully tested with V1 signature.

[Matt Palmer]

Hi,

You may find rfc 1740 useful.

https://tools.ietf.org/html/rfc1740#appendix-A

It contains summary descriptions of AppleSingle and AppleDouble formats.

Regards

Matt

[Andrea Byrne]

Hiya,

Just wanted to add that I wrote about my experience writing this file format signature on OPF here: http://openpreservation.org/blog/2016/09/08/making-the-switch-from-user-to-user-and-contributor-my-first-file-format-signature/

[Tyler Thorsted]

Andrea,

Did you ever look closer at the MacBinary formats? I am looking into AppleSingle and MacBinary as a preservation solution for Macintosh Classic file with resource forks. Ai am also interested in discussing these formats as container so we can identify the format within better.

Happy to discuss.

Tyler Thorsted

[Andrea Kathleen Byrne]

Hi Tyler,

If I recall, I think I looked into creating a signature for AppleDouble, but for some reason, I never did it. I don't remember why .... (my memory and stuff is a bit messed up because of a brain injury last year and I don't know where I put my documentation that's not online about the format). I'm sorry I can't be more helpful. I'd be happy to talk about it once I get up to speed again about what I did. Sorry!!!!

Andrea

--
You received this message because you are subscribed to a topic in the Google Groups "droid-list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/droid-list/G35r2kwtPCY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to droid-list+...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/droid-list/04525178-4265-4564-8ba6-b855ed6bda51n%40googlegroups.com.