If make is GNU make (try make --version to find out) use:
make sudo make install
If gmake is GNU make (i.e. FreeBSD, try gmake --version to find out) use:
gmake sudo gmake install
----- Patch has been tested and is known to work correctly on Mac OS X, Ubuntu and FreeBSD.
----- After building an xar tool with signature support, an xar archive (including Apple's .pkg flat-file format) can be signed like so (example signing commands written for sh-compatible shell):
1. Assume the leaf certificate in DER format is stored in leaf.der and the CA intermediate certificate that issued that leaf certificate is stored in DER format in intCA.der. Also assume the archive to sign is archive.xar and the RSA private key for leaf.der is stored in PEM format in key.pem.
2. Determine the signature size (depends on the RSA private key bit size):
NOTE: only the leaf signing certificate must be given (it should be the first --cert-loc option), zero or more additional --cert-local options may be used to embed intermediary certificates that lead up to the trusted root certificate that the client is expected to already have installed.
A1. Sorry, meant to remove the xar_debug_makefile.patch patch item from the instructions. It's not needed unless you want to run the debugger. You can get it from Bug 75 at:
where it's named "Makefile.inc.in-patch.txt" instead of "xar_debug_makefile.patch". I've also attached it here as "xar_debug_makefile-patch.txt". But again, it's not needed for the signing patch to work.
A2. '3021300906052B0E03021A05000414' has a very specific meaning. See:
In section 9.2 scroll on down to the "Notes." and in note "1." see the value for SHA-1.
A3. The file that's located at the listed defacer URL above now has a correct checksum (and the output of --dump-header is different than what you have listed.)
When you see "Checksums do not match!" it means the checksum of the raw table-of-contents data does not match the checksum stored in the xar file at the offset listed in the table of contents.
You can check this by dumping the header, adding together the "size:" and the "Compressed TOC length:" values and then dumping 20 bytes at that offset in the xar file (assuming the checksum offset in the header is 0 which it almost always is otherwise you have to add that value too).
Compare that sha1 hash value to the value generated by doing:
xar --dump-toc-raw=rawtoc.dat -f some_xar_file.xar openssl sha1 < rawtoc.dat
(FYI, you can get the checksum out of the .xar file using something like:
and then run xxd checksum.dat to see it so you can compare it to the value generated by the openssl command. )
One other thing, when OS X is dealing with xar files if something's wrong with the checksum or the signed checksum differs all you seem to get for an error is something like "could not open" rather than useful information such as bad checksum or bad signature etc.
mackyle, thank you so much for taking the time to answer; let's see if you feel like wasting a bit more time with me!
Perfect for A1 and A2.
Concerning my checksum error, you did clarify the magic behind it perfectly, but I still get the error on any signed '.safariextz' package (including a fresh download of Defacer). I attach another example coming from http://www.macosxtips.co.uk/keysearch/download Any command (-t, -x) will get me the error; the standard ubuntu xar 1.5.2 would open the package no problem. Just to be sure, i'm using three different machines to have a clean xar 1.5.2 and my own compiled xar 1.6dev with and without patch.
Since you seem to be able to work on those archives with no problem I suspect I might be having some issue with my build... It has actually nothing to do with your patch. making again from the 232 source produces the same issue.
Any suggestion on what I might have gotten wrong? Missing some libraries, configure options,...? I'm willing to try ;-) . Or should I just open a new Issue?
PS. I do not have a --dump-toc-raw option, only --dump-toc ... Any other way to extract it?
That's what I get for the checksum (both with xar 1.5.2 and xar 1.6dev)
It's possible you have a conflict between the new xar and the xar package you have installed. I have successfully built and tested the patched xar on both Ubuntu 10.4 i386 32-bit and Ubuntu 10.4 x86_64 64-bit and it works fine.
The xar build creates a library (libxar.so.1 on linux) and an executable (xar). If you were to build the new executable and run it against the old library you would have problems similar to what you've described.
Check the libraries your new xar binary will load with:
ldd xar
You must be having problems building/patching/installing the signature-capable xar because "--dump-toc-raw" is a new option provided as part of the signature patch attached to this issue.
You can check to see you are running the correct xar by looking at the output of "xar --help". If you see "--dump-toc-raw=<filename>" in the help you have the correct binary. If you then run "ldd xar" on that binary and see that it's really loading the newly installed library (should be /usr/local/lib/libxar.so.1 unless you used a --prefix option on configure) then you should be good to go.
The only configure option you need is the one shown above "--without-lzma" because the xar sources don't seem to compile against current lzma libraries.
In case you are having trouble applying the patches, I've attached an archive of the patched sources that have also already had "./autogen.sh --noconfigure" run on them. You should be able to build and install by doing:
tar xvzf xar_sig_232_src.tar.gz cd name-of-newly-extracted-directory ./configure --without-lzma make sudo make install
mackyle, thanks once again for trying and helping.
Indeed I do have --dump-toc-raw, my bad; in trying and understanding my checksum error I had recompiled xar /without/ the patch, and expected to find dump-toc-raw nevertheless.
So: the checksum is fine.
For: xar --dump-toc-raw=keysearch.rawtoc.dat -f keysearch.safariextz openssl sha1 < keysearch.rawtoc.dat
I do get the same 5f73c611ce09b518c92edd4ca084f4b83ac54a14 I got before, but
xar -tf keysearch.safariextz
still gives the same
Checksums do not match! Error opening xar archive: keysearch.safariextz
error as before.
Hoping I'm not bothering you too much, any idea what this could be due to?
I tryed uninstalling any xar and building from your code, but I get exactly the same error!
kukushechkin, does this simple thing work for you? Can you open with xar a .safariextz package?
Also! I compared .safariextz, generated by Safari and manually, both have equal checksum, but it differs between them, which seems odd to me as they were signed with the same certificate.
I know just built a new xar on yet another ubuntu 10.4 and this time I have the checksum error no more. Good, though I'd still like to understand what was wrong on the other machine. I had to build errors.
So I generate a signed safariextz and got the same error I started with "Safari can't install this extension. An error occurred while installing this extension.". I figure the generated xar package is somehow not compliant with apple one, and know I do not have to worry about checksum errors on my side any more.
In case it might help, I attach my test:
* the 'keysearch' (I did not develop it, it's just a test!) extension that I extract, re-package with xar and sign with my certificate
* the 'keysearch' extension re-signed with my certificate using safari extension builder
Differences I can spot in the TOC: * I did not include in my signing the root certificate (might that be an issue?) * there's a lot of meta generated by my xar that there is not in the apple one (all file attributes: time, owner, etc.)`
and with that different of course also byte 20 to 256 are different (am I right assuming that's where the signature is stored?)
I now just built a new xar on yet another ubuntu 10.4 and this time I don't have the checksum error no more. Good, though I'd still like to understand what was wrong on the other machine. I had no build errors.
So I generate again a signed safariextz and got the same error I started with: "Safari can't install this extension. An error occurred while installing this extension.".
I figure the generated xar package is somehow not compliant with apple one, and know I do not have to worry about checksum errors on my side any more, so I can explore some more.
In case it might help, I attach my test (and btw I did not develop keywordsearch, it's just a test!) :
* the 'keysearch.signed' extension that I extract, re-package with xar and sign with my certificate
* the 'keysearch.xbsigned' extension re-signed with my certificate using safari extension builder
Differences I can spot in the TOC: * I did not include in my signing the root certificate (might that be an issue?) * there's a lot of meta generated by my xar that there is not in the apple one (all file attributes: time, owner, etc.)`
and with that different of course also byte 20 to 256 are different (am I right assuming that's where the signature is stored?)
does this help?
mackyle have you ever been able to specifically signing a safari extension and have it accepted?
I have found and corrected some bugs in the xar_from_base64 function. A new patch is attached as xar_cli_signature_232v2-patch.txt and, for convenience, a fully patched and autogen'd set of sources ready to be ./configure'd as xar_sig_232v2_src.tar.gz.
The updated patch has the following fixes:
* xar_from_base64 no longer adds extra bytes to the decoded output (longstanding bug) * the version number output by xar --version is now 1.6dev_s1 to distinguish from earlier patches * the extra blank line that could sometimes be inserted in the --extract-CAfile output certificates has been removed
I have also examined the files attached to Comment 10. The keywordsearch.signed.safariextz file contains an invalid certificate. It had one byte of garbage appended to it -- precisely the bug that the xar_from_base64 function had before the attached, updated patch.
Did you check to see that you had a valid certificate by running "openssl x509 -noout -text -inform der -in mycertificate_file" before you passed it to xar with the --cert-loc option?
The xar_from_base64 bugs did not affect signing, but they did affect the output of --dump-toc, --extract-certs and --extract-CAfile. Did you perhaps extract the certificate used to sign keywordsearch.signed.safariextz by running one of the --dump-toc, --extract-certs or --extract-CAfile options on keywordsearch.xbsigned.safariextz (or another signed xar) and then use that to sign keywordsearch.signed.safariextz? That would explain how you ended up with a certificate with one byte of garbage data appended to the end.
FYI, you can easily extract your certificate from the Mac's Keychain Access by selecting the certificate and exporting it as ".cer" format (which is really the binary DER format that you need to pass to the --cert-loc option).
> did you try to add root certificate? And how do you obtain .pem and .der?
(same question from mackyle and kukushechkin)
I haven't tried adding the root certificate yet; I extracted/converted my certificate and key using OpenSSL from a windows pfx (I could have gotten a DER directly but not the pem key)
> Did you check to see that you had a valid certificate
The certificate looks fine to me (see the attached file), and I did not use any xar function to extract the certificate; but I will definitely run your patch and try everything once again. It might not be today, but I'll do it ASAP.
I am looking for a way to create a .safariextz from a build system on linux. I also tried this patch with same result as kukushechkin. I did comparisons to what Safari produces and verified I have the same certificates. Any help is appreciated.
I've created a new patch file xar_cli_signature_232v3-patch.txt and, for convenience, a fully patched and autogen'd set of sources ready to be ./configure'd as xar_sig_232v3_src.tar.gz.
The updated patch has the following fixes from crckyl's patch:
* lib/util.c now does "#define _FILE_OFFSET_BITS 64" which should help with checksum mismatch issues * lib/signature.c now generates a signature-creation-time element to hopefully make Safari happy
And also contains the following additional changes:
* cleaned up some of the help in the xar --help output and xar man page * xar --version now outputs "1.6dev_sigv3" to distinguish this patch from the earlier ones
If you really want to try and duplicate a .safariextz file you should start with a .safariextz file that has been signed by Safari's Extension Builder and extract the embedded certificate chain from it for use in the xar --sign command.
For example, suppose you had a Safari Extension Builder signed extension named "keywordsearch.xbsigned.safariextz" then you would do the following:
1. mkdir certs 2. xar --extract-certs=certs -f keywordsearch.xbsigned.safariextz
The certs directory will now contain each certificate in the chain that was embedded into the .safariextz xar file. The first file "cert00" should be identical to your Safari signing certificate you got from Apple (Safari appears to be hard-coded to only accept extensions signed by Apple-issued certificates so don't bother trying to roll your own for development purposes).
If you perform the above two steps on the same-named file that was attached to comment 10 above, you will end up with a cert00, cert01 and cert02 file in the certs directory. cert00 will be the leaf signing certificate (provided to you by Apple), cert01 will be an Apple intermediate certificate and cert02 will be an Apple root certificate.
All these certificates other than cert00 should be specified on the "xar --sign" command line in numerical order (using a --cert-loc option for each) AFTER the --cert-loc option that specifies the Apple-issued certificate.
For example, in comment 14 above, step #5 would be changed to:
Note how cert00 IS NOT specified since it's presumably a duplicate of "key.der" (and therefore you could actually specify certs/cert00 INSTEAD OF key.der).
Also note that the number of certificates produced by the --extract-certs command may vary -- it depends on how many are actually embedded in the xar file archive -- there will always be at least one if the archive is signed (extracted as cert00). The "xar --sign" command will need to be adjusted by adding or removing "--cert-loc" options accordingly depending on how many certificates are produced by the "--extract-certs" option.
This gets me further than the previous patches, to the point where if I open the signed .safariextz file Safari opens and asks if I want to install it. However, if I try to install it, Safari crashes. I can still install the Extension Builder-built extension just fine, however.
I can't seem to find any meaningful output as to why it's failing, other than:
Here's some more output from Safari when it crashes from the console log:
Safari(22604,0x7fff711a1ca0) malloc: *** mmap(size=158329674399744) failed (error code=12) *** error: can't allocate region *** set a breakpoint in malloc_error_break to debug
You might try adding the "--distribution" option to the xar command line when you build the initial xar archive to omit extraneous attributes that you don't really need. This should hopefully prevent any significant use of the xar_darwinattr_extract function when Safari attempts to install the extension.
In other words, if your unpacked extension is in the MyExt.safariextension directory, use
xar -cvzf MyExt.safariextz --distribution MyExt.safariextension
Also after you've built your signed .safariextz, try extracting it using xar (i.e. xar -xvf MyExt.safariextz) to make sure the archive isn't corrupt.
I can't really tell what caused the above crash, but you might try looking at this output:
xar --dump-toc=- -f MyExt.safariextz | grep "<size>"
and see if you see any negative size values (attempting to extract the archive should also expose any negative size values as well) as it looks like from the code that if a size value was somehow negative you might see a crash. Hopefully though the "--distribution" option will solve the problem for you.
After compiling xar_232v3_src.tar.gz with the patchs xar_cli_signature_232v3-patch.txt and xar_debug_makefile-patch.txt, I got:
:$ xar --version xar 1.6dev_sigv3
And with a super simplified process:
1. mkdir certs 2. xar --extract-certs=certs -f someExtension.safariextz (here I compared the three cert0x with the certs in the keychain and they are exactly the same)
3. Then with the certificates I can in theory do something like this: xar -c dir_with_my_extension_stuff/ -f newExtension.safariextz --sign --sig-size 256 --cert-loc cert00 --cert-loc cert01 --cert-loc cert02
but I get the very informative message "Safari can't install this extension\n An error ocurred while installing this extension"
If I do something like "xar --dum-toc=somefile -f " with both the original and the new xar file, I see some little differences, the main one in the tag offset, in the new file the offset is bigger by 15 bytes. The other differences are that in the new xar file some extra information related to files like inode, deviceno, user, mode and blah blah, are listed but not for directories. In the original xar, is the opposite, this kind of info is listed for directories but not for files. I tried the option --distribution and it fixed the offset, and removed almost all the extra info, and only left behind the "mode" tag. With --distribution the only differences are extra "mode" tag and the lack of information tags about directories.
And the error still remains.
Some Ideas? By the way the certificates from the keychain are ".cer". And I obtained the size from the original file header itself (--dump-toc), There you can see the offset (20), the size (256), siganture style (RSA), they all match with the new xar file.
And "openssl dgst -sign key.pem -binary" did not worked on any attempt. I tried different ways like converting a cer into a pem or extracting the certificates as a pem, but no success. So, the size from the original header became very useful.
If you read the man page, "--sign Creates a placeholder signature and saves the data to sign to disk." Steps #4 and #5 (from the top of this bug) are not optional. xar does not understand how to create a real signature (it only understands how to manipulate xar archives), so the placeholder still left in the file because you skipped steps 4 and 5 does not verify and Safari refuses to acknowledge the file since it has an invalid signature.
"key.pem" is your private signing key. It should be in your keychain. Use Keychain Access to export it as a .p12 file. Say it's called "my_private_key.p12" then you use the following to convert it to PEM format:
And now you have a private key in key.pem to use for steps 4 and 5. Don't forget to re-add the --data-to-sign option to step 3 or you will have nothing to sign.
