PDF Background / annotation

[Stefan Langerman]

Hi all,

For myself and many people around me, the most needed feature in Quill would be the possibility to use it to annotate pdfs.

I've been looking into how to make it work but I'm still not too familiar with the Quill code (and android in general.

It seems like there are two things we have to be able to do:

1. Display the pdf as a background to the pages

2. Save the Quill strokes as annotations on top of the pdf.

I could not find a single library that can do both tasks

for 1. several pdf viewers use mupdf (ebookdroid, APV)

for 2. http://itextpdf.com/ is able to annotate existing pdfs

For the behavior, I would imagine creating a notebook from a pdf, which would automatically create one page for each page of the pdf file.

Deleting a page should delete both the page and the annotations on it, inserting a page should insert a blank page, duplicating a page should duplicate the pdf background as well. If we want to be more fancy later, we could add the possibility to set the background of any page of a notebook to any page of any pdf.

For now, it seems the main challenges will be: creating pages with the correct aspect ratio, interface with mupdf to render the pdf in the background, ...

What do we do if the user tries to change the aspect ratio of a page which has a pdf background?

Any pointers to parts of the code I should look at more closely? Better libraries?

Stefan.

[Lars Wallin]

Hi,
This would be similar to my idea, which is to make it possible to import custom graphic as background (page type). My suggestion was to use svg, but pdf makes more sense perhaps.

/Lars

[Lars Wallin]

For now, by the way, you can use ezPdf Reader Pro on Android for annotations :)

On Feb 10, 2012 11:33 PM, "Stefan Langerman" <s...@slef.org> wrote:

[Volker Braun]

First of all, we have to distinguish PDF annotations (that you can click on to show/hide in the PDF) from a flat PDF with embedded pen strokes that can't be turned off

Personally, I would be more interested in the second approach. The problem is how to combine the two PDF streams. Its not an insurmountable problem, but there is no existing GPL library that I know of that just does it. It wouldn't be too hard to write one, though. I'll probably do it eventually if nobody else does it first ;-)

[Stefan Langerman]

xournal for linux does this quite well, maybe we should look in their code to see how they do it.

Otherwise, what is the issue with using annotations?

[Volker Braun]

From my own experience, I once "signed" a PDF form with a graphical annotation and sent it back to the secretary. She promptly told me that the signature doesn't show up on the hardcopy. While you certainly can print the annotated file, its not entirely obvious. Try to explain to your secretary the difference between annotations and normal PDF objects :-)

[Stefan Langerman]

good point.

It might actually not be a bad idea to give both options...

It seems like iText can do both these things. Am I right?

See the example here: http://itextpdf.com/examples/iia.php?id=113

[Volker Braun]

Is the Android version of iText also AGPLv3 licensed? I've seen various accounts. On a related note, does iText on Android support all of these features? If you search around then there are some java.awt dependencies that are not part of Android dalvik.

[Stefan Langerman]

I cannot find that info... or even find the library specifically for Android. I think it might not be released yet:

They announce that it will appear in future versions: http://lowagie.com/itext510

But he also explains how he built it from the core library: http://lowagie.com/itext-android-prototype

So if we do the same (mainly removing references to awt), the resulting library will be  AGPLv3.

He will talk about this in the iText summit in Belgium!

http://lowagie.com/summit2012-BL1

Maybe I'll go.

[usoban]

I've had experience using two different PDF readers that were able to add content to existing file.

The first one is HTC's default PDF Viewer, which is able to annotate PDF directly with a pen and highlight text. The second is Adobe's PDF Reader, which has more annotating and highlighting options, such as underlining, strike-through, and appending a note on a text, but lacks the ability to work with pen directly (one needs to use the toolbar buttons to activate the pen first, at least that's the situation with 3.2)

I do agree that having an option of flattening a document is crucial, especially for singing documents. However, without having editable annotations, the annotating could prove quite useless. So in my opinion the solution would be having both modes. E.g., HTC's PDF Viewer solves that at save time, offering you both options.  

I'm willing to look more into this, as I've not found an ideal app yet, I'm tired of using both of them, and I really like the idea of 'inserting' a page :)

[Volker Braun]

Let me paste some thoughts about PDF reader/annotation functionality that I wrote a while ago. Sadly I didn't have any time to implement them, but here they are:

I think this PDF annotating/editing a logically different task, so it would be awkward to try to stuff it into Quill as well. But of course a lot of Quill code could be reused, and I've worked towards separating things into reusable components. To do work with PDF files, one probably wants to use

  * muPDF to render the PDF. Works on ARM and other projects have written JNI wrappers

  * libHaru to generate pdf for the written layer (this is what Quill uses)

  * port pdftk to ARM and use it to merge foreground (handwritten) layer with the original pdf stream.

[Stefan Langerman]

Hi,

I just posted a first very buggy proof of concept for pdf background, see http://code.google.com/r/sl-android-quill/

To run it, you have to include APV in your project http://code.google.com/p/apv/ 

I did this because I was too lazy to recompile mupdf myself. The drawback is APV gets installed along with Quill. There is probably a better solution.

The update lets the user open a pdf with Quill, and creates a new Book with all the pages each with a corresponding pdf background.

Many things to fix (right now, adding a pdf makes quill rash when trying to switch notebooks -- but this is mainly a proof of concept) : 

- need to change the save format to save the pdf info (for now, 1 pdf per Book, the pages contain background pointing to the pdf and the page. We might want to plan for the possibility of including pdf backgrounds from several pdf files in a same book.

- The zoom level for pdf rendering should probably be increased, and the code could be modified to render only the visible part of the page (maybe using Tiles to cache rendering as in APV? although rendering is quite fast, we might not need that.)
- Should maybe store a hash of the pdf and if the same pdf is opened, offer to just go to that book or create a new one

- Need to fix the aspect ratio business. Mupdf gives the document size in in/72. We should allow creating new aspect ratio objects if necessary, but we probably don't need one per page. Maybe adding a few more aspect ratio definitions would do the trick...

- Should allow to assign pdf background on any page using the paper dialog (?)

- Find the right behaviour for add page/ duplicate page etc.

- ...

And then of course start working on the pdf output, but that can wait for now.

What do you think?

Stefan.

[Volker Braun]

Sounds great!

My plan was to have arbitrary binary objects saved with the notebook, identified by UUID. Images are so far the only example, but I had in mind that this is extendable to backgrounds / audio / etc.

The aspect ratio can be any floating point number and are saved internally as such. The predefined aspect ratio objects are just for the GUI. 

I think it would be generally useful to have mupdf jni bindings as an independent Android library so that not everybody has to reinvent the wheel. In principle this should be easy, but I think that in practice this'll need some love to correctly deal with all kinds of pdf files that one might encounter. And not crash&burn when facing invalid/broken pdf files.

Best,

Volker

[Stefan Langerman]

On Saturday, December 22, 2012 3:12:57 AM UTC+9, Volker Braun wrote:

Sounds great!

My plan was to have arbitrary binary objects saved with the notebook, identified by UUID. Images are so far the only example, but I had in mind that this is extendable to backgrounds / audio / etc.

Is that the "blobs"?

In general, I think it would be good to not save the pdf background with the notebook, as 

- pdf files can be large and 

- you might want to keep both the original and the annotation, and 

- you might need to use the same pdf as a background many times (e.g. forms or just as an easy way for the user to define custom paper)

Internally, maybe it might be good to have a store for some objects that can be shared for all notebooks (such as pdfs or images), or maybe even allow to keep some resources external. We can just generate a hash for each pdf and store that along with its location, or automatically make an internal copy if it is not too large, but have a local store for all hashed resources.

For an output format, I actually really like the way xournal does it: it never touches the pdf background, and a .xoj file is just an xml file linking to the pdf background and describing the strokes and other objects on the pages.

Another file format option was suggested to me by the author of Ipe http://ipe7.sourceforge.net/ : his file format is pdf. He saves the file as a pdf file that contains the drawing and adds xml information about all the objects created as metadata so the file can be reopened and parsed by Ipe.

  

The aspect ratio can be any floating point number and are saved internally as such. The predefined aspect ratio objects are just for the GUI. 

Yes, but in the Background code it takes the closest match in the official list of aspect ratios so I had to change that to allow arbitrary aspect ratios.

 

I think it would be generally useful to have mupdf jni bindings as an independent Android library so that not everybody has to reinvent the wheel. In principle this should be easy, but I think that in practice this'll need some love to correctly deal with all kinds of pdf files that one might encounter. And not crash&burn when facing invalid/broken pdf files.

Yes. That is why I used the APV package. It seems to be regularly maintained so it seemed easier than recompiling it all myself.

Note that mupdf does include JNI bindings for Android in its source package. Actually they include a sample app and instructions on how to compile, see http://git.ghostscript.com/?p=mupdf.git;a=tree;f=android;hb=HEAD . So maybe making an independent android library is just a matter of removing the sample app? Or do the same with APV?

EBookDroid seemed like a good option too. It is well maintained and supports other formats such as djvu and some ebook formats (it would be fun to allow arbitrary annotations on books... maybe then it is even more important not to duplicate the original file which is sometimes huge).

APV just seemed easier to start with for now.

[Volker Braun]

Yes, blob = http://en.wikipedia.org/wiki/Binary_large_object

I would prefer to include any pdf files (or perhaps extract individual pages used and include these). Otherwise there is no good way to move notebooks between devices or to your desktop. I agree that the pdf shouldn't be modified otherwise, that is, only create a new pdf when you export. And, compared to images, pdf files are comparably small.

Speaking of Xournal, we now have a quill<->xournal converter if you haven't noticed: 

https://github.com/vbraun/QuillDesktop

[Stefan Langerman]

Well 1-page pdfs that contain mostly text are small, but scanned books/articles or pdfs with many images can take several hundreds of megs.

So maybe storing the pdfs internally is ok, but then maybe we should try to identify big objects by checksum application-wide to avoid wasting too much space (for images too).

Anyway, I just pushed my last updates. Load/save is fixed (increasing the file version number) and resolution has been improved. 

Somehow there is still a problem with APV/mupdf: the first time a pdf is loaded it works perfectly, however the second time the same file is loaded (even after a reboot), quill crashes with a signal 11, which I have traced back to a call to the PDF library. Either we have to clean something up explicitly or we'll have to try a different library maybe...

Btw, is there a way to get the area of the canvas that needs to be drawn in Background.draw? Right now I am setting the zoom to some arbitrary amount that seems to be sufficient but it would be nicer to compute the right zoom and maybe even only draw the part of the canvas that needs to be drawn.