Implementing PDF 1.5 object streams and xref streams
Maik Musall
I'm working on a pdf library that can read, write and modify PDF files.
Adobe made us some work updating it to deal with the PDF 1.5 extensions,
especially object streams and cross reference streams. I think I've got
it complete now, but applications don't respond as I expect.
I made a test by extending the PDF Reference Manual in 1.5 format (as
available from
http://partners.adobe.com/asn/acrobat/sdk/public/docs/PDFReference15_v6.pdf,
9190216 Bytes, md5sum 7f47955c825ed83dfe2a6a2fac1a092) with an update
section that places a large string on the first page. But the string
doesn't get displayed in xpdf, and Acrobat talks of a damaged file.
I'm stuck in development because I don't know what's still wrong. I
created an object stream that /Extends the correct existing one, wrote a
xref stream containing all new and updated objects, and wrote a new
startxref number pointing to the xref stream, and my own code is able to
correctly read this file in again :-)
I know this is a question quite "deep inside", and most people won't be
able to answer this, but if you think you can, I would be glad for every
small hint. I don't want to post binary data here, and don't want to
publish the whole 9MB file, but you probably have the PDF reference
manual already. You can get the updated section at
http://www.musall.de/files/rest.pdf. Just concatenate it to the above
mentioned PDF reference file. The updated section is just 1900 bytes.
Streams are uncompressed so that stream objects and xref streams are
directly viewable with any hex editor.
Thanks and regards
Maik
Dmitry Sklyarov 291875429
You wrote on Mon, 19 Apr 2004 18:54:53 +0200:
MM> I know this is a question quite "deep inside", and most people won't be
MM> able to answer this, but if you think you can, I would be glad for
MM> every small hint.
You have specified invalid /Size value in new xRef stream dictionary.
Correct value is not 56594 but 57397.
With best regards, Dmitry Sklyarov 291875429. E-mail: dm...@elcomsoft.com
Maik Musall
Thanks a lot for that hint! I was blind for that because /Size was
initially correct, but then I changed something in the code that
overwrote the correct value with the one from the original file.
Maik
Maik Musall
Dmitry Sklyarov 291875429 wrote:
> Hello, Maik!
> You wrote on Mon, 19 Apr 2004 18:54:53 +0200:
>
> MM> I know this is a question quite "deep inside", and most people won't be
> MM> able to answer this, but if you think you can, I would be glad for
> MM> every small hint.
>
> You have specified invalid /Size value in new xRef stream dictionary.
> Correct value is not 56594 but 57397.
Although 56594 is of course wrong, I think 57398 is right, not 57397,
because the highest object id I use is 57397, and the /Size entry has to
carry the number one higher than the highest object number.
Nonetheless, it's not yet working. If you change that characters in
rest.pdf and concatenate it to the spec file, Acrobat6 will still not
open it but report an error code 131 - which leads me to the problem
that I don't have a list of those error codes above 126. 131 Seems to be
one that's new to Acrobat 6, and I couldn't find a list of those codes
anywhere.
Maik
Maik Musall
>> You have specified invalid /Size value in new xRef stream dictionary.
>> Correct value is not 56594 but 57397.
>
> Although 56594 is of course wrong, I think 57398 is right, not 57397,
> because the highest object id I use is 57397, and the /Size entry has to
> carry the number one higher than the highest object number.
>
> Nonetheless, it's not yet working. If you change that characters in
> rest.pdf and concatenate it to the spec file, Acrobat6 will still not
> open it but report an error code 131 - which leads me to the problem
> that I don't have a list of those error codes above 126. 131 Seems to be
> one that's new to Acrobat 6, and I couldn't find a list of those codes
> anywhere.
I found another bug, the xref stream didn't include an entry for itself.
I replaced http://www.musall.de/files/rest.pdf with a new version
containing the fixes but still throwing the above mentioned error 131.
Maik
Dmitry Sklyarov 291875429
You wrote on Wed, 21 Apr 2004 09:57:33 +0200:
MM> I myself wrote:
??>>> You have specified invalid /Size value in new xRef stream dictionary.
??>>> Correct value is not 56594 but 57397.
??>>
??>> Although 56594 is of course wrong, I think 57398 is right, not 57397,
??>> because the highest object id I use is 57397, and the /Size entry has
??>> to carry the number one higher than the highest object number.
Agreed.
??>> Nonetheless, it's not yet working. If you change that
??>> characters in rest.pdf and concatenate it to the spec file, Acrobat6
??>> will still not open it but report an error code 131 - which leads me
??>> to the problem that I don't have a list of those error codes above
??>> 126. 131 Seems to be one that's new to Acrobat 6, and I couldn't find
??>> a list of those codes anywhere.
Try press Ctrl-Enter when you get error message in Acrobat/Reader.
Specifically for error 131 explanation text is "Error in XRef stream."
MM> I found another bug, the xref stream didn't include an entry for
MM> itself. I replaced http://www.musall.de/files/rest.pdf with a new
MM> version containing the fixes but still throwing the above mentioned
MM> error 131.
In XRef stream you have two entries with type 2 (reference to compressed
object).
For both entries index = 1 is specified, but object streams referred
contains one object each.
Change index value to 0 in both cases.
With best regards, Dmitry Sklyarov. E-mail: dm...@elcomsoft.com
Maik Musall
> Try press Ctrl-Enter when you get error message in Acrobat/Reader.
> Specifically for error 131 explanation text is "Error in XRef stream."
That's a good hint. Works also in Acrobat 6 Professional. Thanks.
> MM> I found another bug, the xref stream didn't include an entry for
> MM> itself. I replaced http://www.musall.de/files/rest.pdf with a new
> MM> version containing the fixes but still throwing the above mentioned
> MM> error 131.
>
> In XRef stream you have two entries with type 2 (reference to compressed
> object).
> For both entries index = 1 is specified, but object streams referred
> contains one object each.
> Change index value to 0 in both cases.
Yup, that was it. Now it opens correctly. Perhaps Adobe should be more
clear in the spec that the index starts at 0 in that case.
What however still puzzles me is that while Acrobat [Viewer] 6 opens the
file and displays the new content on the first page, xpdf 3.00 (which is
my primary viewer since Acrobat Reader 6 is not yet available on Linux)
opens the file but does not display the content. I tried the same with a
non-linearized version of the same document; with that xpdf even refuses
to open the file but complains about not being able to process the xref.
Since Acrobat 6 doesn't complain about anything in both cases, I suspect
xpdf lacks correct implementation of update sections with xref streams.
I'll contact the author on that.
Thanks for your valuable help, Dmitry!
Regards
Maik