Idea - use google docs viewer for doc, pdf etc display

[Vasik Greif]

Hi, I was thinking about displaying the documents inside RS in a more
user friendly way, and came across the google docs viewer idea. It
works really well for online files, you just pass the url to the
adress, http://docs.google.com/viewer?url= , but for displaying the
local files, one would have to start http server to be able to open it
in the viewer. There's a python class here, which does exactly that
http://www.youtube.com/watch?v=SFi5Plnu3aU, but a php solution would
be better for sure.. Anyone likes this?

[Vasik Greif]

Probably not so good idea because of security and google caching the
file...:( Any better ideas?

On 18 ún, 13:49, Vasik Greif <vaclav.gr...@gmail.com> wrote:
> Hi, I was thinking about displaying the documents inside RS in a more
> user friendly way, and came across the google docs viewer idea. It
> works really well for online files, you just pass the url to the

> adress,http://docs.google.com/viewer?url=, but for displaying the

> local files, one would have to start http server to be able to open it

> in the viewer. There's a python class here, which does exactly thathttp://www.youtube.com/watch?v=SFi5Plnu3aU, but a php solution would

[mrpatulski]

Has anyone looked at the Internet Archive BookReader as a solution?

The BookReader was developed by the Internet Archive and open source contributors to provide online access to scanned books. The Internet Archive has more than 1,000,000 scanned books available to read online. The BookReader is also used to provide access to materials from many other organizations. The BookReader source code is available in the openlibrary.org github repository.

This landing page has mor details: http://openlibrary.org/dev/docs/bookreader

[Vasik Greif]

That looks great! I have no idea how that works/how to implement
this...???

On 18 ún, 14:40, mrpatulski <matt...@patulski.is> wrote:
> Has anyone looked at the Internet Archive BookReader as a solution?
>

> *The BookReader was developed by the Internet Archive and open source

> contributors to provide online access to scanned books. The Internet
> Archive has more than 1,000,000 scanned books available to read online. The
> BookReader is also used to provide access to materials from many other
> organizations. The BookReader source code is available in the

> openlibrary.org github repository.*

[John HOLMES]

In a somewhat similar vein, I was thinking of using YouTube for video.  YouTube supports private, or by encrypted url only (much like how RS generates unique URLs for sharing a collection).

We are a K-12 school and so:

1. Bandwidth is an issue
2. server space/backup is an issue

We are using RS for students and teachers alike.  We've only just started a one laptop/child in grades 6 and 9 this year, and we officially launched RS last month as a means for members of our community to share resources they've created. They can then use this content in Moodle (open source learning management system), or Google sites, blogs, wikis etc. 

In August we roll out laptops for all kids grades 6-12.  Already we're seen RS take off, and so I'm quickly looking at ways to scale.

I believe that in many cases the greatest number of hits a resource will receive will be in the first few months after creation (especially true for student created content).  I've been thinking that on upload, we run the YouTube API (at night when bandwidth ins't at a premium), and duplicate/backup all videos to our school YouTube channel, with access set to private.

Then when the resource is 'found' in RS, we check if:

1. address is external to our school.  If yes, serve from YouTube
2. if device requesting is iOS then serve from YouTube (since html 5 is not yet supported by RS), a nice bonus
3. else serve from our server.
   

Then using some algorithm (ex. 1 month since last view and views less than 10) , we could delete the resource from the filestore (believing most hits have already taken place) and then just serve it from YouTube.

This also acts as our backup as all video content would be on YouTube in the event of a catastrophic failure. (we run RAID)

In a similar vein, Google Picasa API could be used for photos.  Storage is unlimited for photos 2000x2000 px (provided you've joined Google+).  For almost all of our purposes this image size is sufficient for our uses.

Clearly this will take some customization, but we have two php developers at our school who I think could handle it.

I'd be curious as to peoples thoughts/first reactions.

John

Jakarta International School

[Vasik Greif]

The YouTube part sounds great!

I would suggest to do it as configurable as possible - ie. upload
videos with "youtube" keyword only, add private/public option
(keyword based too, probably), option to set the field for the name,
description and keywords etc. This would be great function for me, as
now I have to upload videos to RS and YT manually...

For the Picasa, I don't see any use of it for me, except for the
"backup". For the space issue I did a plugin for resizing the large
originals - se here: http://resspace.blogspot.com/2012/02/resourcespace-resize-original.html

Vasik

On 18 ún, 16:32, John HOLMES <jhol...@jisedu.or.id> wrote:
> In a somewhat similar vein, I was thinking of using YouTube for video.
>  YouTube supports private, or by encrypted url only (much like how RS
> generates unique URLs for sharing a collection).
> We are a K-12 school and so:
>

>    1. Bandwidth is an issue
>    2. server space/backup is an issue

>
> We are using RS for students and teachers alike.  We've only just started a
> one laptop/child in grades 6 and 9 this year, and we officially launched RS
> last month as a means for members of our community to share resources
> they've created. They can then use this content in Moodle (open source
> learning management system), or Google sites, blogs, wikis etc.
> In August we roll out laptops for all kids grades 6-12.  Already we're seen
> RS take off, and so I'm quickly looking at ways to scale.
>
> I believe that in many cases the greatest number of hits a resource will
> receive will be in the first few months after creation (especially true for
> student created content).  I've been thinking that on upload, we run the
> YouTube API (at night when bandwidth ins't at a premium), and
> duplicate/backup all videos to our school YouTube channel, with access set
> to private.
> Then when the resource is 'found' in RS, we check if:
>

>    1. address is external to our school.  If yes, serve from YouTube
>    2. if device requesting is iOS then serve from YouTube (since html 5 is
>    not yet supported by RS), *a nice bonus*
>    3. else serve from our server.

[David Dwiggins]

I glanced at this a while back. I think it has a lot of potential, and would like to play with it at some point. 

To get full benefit from it, it might be necessary to do some more thinking about how RS handles complex compound documents like multipage books and PDFs. I suspect we're going to have to put some thought into this at HNE at some point in the future, but it hasn't become a big issue yet.

-David

--
You received this message because you are subscribed to the Google Groups "ResourceSpace" group.

To view this discussion on the web visit https://groups.google.com/d/msg/resourcespace/-/Nm28JrXlrYIJ.

To post to this group, send email to resour...@googlegroups.com.
To unsubscribe from this group, send email to resourcespac...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/resourcespace?hl=en.

[mrpatulski]

@John Holmes,

you raise some good points about video but it really should be another thread--If you re-post, I have details on the RS API and html5 video work we are funding that I can contribute fo the conversation

[mrpatulski]

@David I looked into the Internet Archive BookReader after thinking about upgrading a bespoke document viewer we developed for RS. The additional functionality--image optimization, player skinning, text to voice, embeddable viewer, open standards support, and large developer community makes ABR an attractive feature to consider integrating. Technically what are the big picture pain points?

[David Dwiggins]

It might actually not be that difficult for supporting single-resource PDFs. Since we often photograph and store pages of documents as individual resources, I've been thinking about it partly as having the ability to understand the full structure of a group of resources that are all part of a single book/document. But that may be more than what is needed for a basic implementation that works with PDFs, Word docs, etc.

I will try to look at this again in the next few days and ponder what it would take to get it going. 

-David

On Sat, Feb 18, 2012 at 6:12 PM, mrpatulski <mat...@patulski.is> wrote:

@David I looked into the Internet Archive BookReader after thinking about upgrading a bespoke document viewer we developed for RS. The additional functionality--image optimization, player skinning, text to voice, embeddable viewer, open standards support, and large developer community makes ABR an attractive feature to consider integrating. Technically what are the big picture pain points?

--
You received this message because you are subscribed to the Google Groups "ResourceSpace" group.

To view this discussion on the web visit https://groups.google.com/d/msg/resourcespace/-/R5VToNChB7MJ.

[Robert Whitacre]

how about libreoffice on the server for conversions and what not.   I currently help on a Java/tomcat based system that uses LibreOffice. for file conversions. 

Robert Whitacre
(Web Master of whitacrefamily.com)

[Vasik Greif]

I got lost a little in this thread.

To summarize:
- using google doc's doesn't seem to be good idea because of the
security.
- Internet Archive BookReader looks great, anyone knows how it
actually works and what would be needed to implement it for PDF/doc
display
- group of resources as single book/document - maybe collections with
specific tag could be used for this? Or, an special, "virtual"
resource type?
- libreoffice - I'm pretty sure OpenOffice is used for some
conversion, but I don't know much about it. Anyone?

Vasik

On Feb 19, 11:45 pm, Robert Whitacre <rob...@whitacrefamily.com>
wrote:

> how about libreoffice on the server for conversions and what not.   I
> currently help on a Java/tomcat based system that uses LibreOffice. for
> file conversions.
> Robert Whitacre
> (Web Master of whitacrefamily.com)

>  <http://www.whitacrefamily.com>

>
>
>
>
>
>
>
> On Sun, Feb 19, 2012 at 3:44 PM, David Dwiggins <da...@dwiggins.net> wrote:
> > It might actually not be that difficult for supporting single-resource
> > PDFs. Since we often photograph and store pages of documents as individual
> > resources, I've been thinking about it partly as having the ability to
> > understand the full structure of a group of resources that are all part of
> > a single book/document. But that may be more than what is needed for a
> > basic implementation that works with PDFs, Word docs, etc.
>
> > I will try to look at this again in the next few days and ponder what it
> > would take to get it going.
>
> > -David
>

[Dan Huby]

On Feb 19, 10:45 pm, Robert Whitacre <rob...@whitacrefamily.com>
wrote:

> how about libreoffice on the server for conversions and what not.   I
> currently help on a Java/tomcat based system that uses LibreOffice. for
> file conversions.
> Robert Whitacre

We already support conversions via unoconv - a quick Google search
informs be that this works with LibreOffice.

As this gets converted to PDF then uses the existing PDF functions in
RS, you get the paging capability so can preview all pages online with
no need for a special viewer plugin.

Hope that helps,

Dan

[Vasik Greif]

> As this gets converted to PDF then uses the existing PDF functions in
> RS, you get the paging capability so can preview all pages online with
> no need for a special viewer plugin.
>

Yup, this is right, but the paging is not really user-friendly - the
Archive BookReader has zooming, thumbnail view, easy navigation thru
the document, search inside etc., which are really important features
to a serious documents use...

[Vasik Greif]

Ok, I got the book reader basically working.

http://screencast.com/t/YWZMjfsRMtKD

The biggest issue I have is I have no idea how to tell the reader to
use specified images as a source for the book. So, what I did is I
added a function to preview_preprocessing.php, which creates a
subfolder inside the resource folder, and saves the pages as
sequentially numbered images. Than I point the reader at the folder.
This is not a great way how to do it, but it works and looks great!
If anyone knows how to fetch individual images as the source for the
book, please share it here, as that would be much better way for sure.

Vasik

[John HOLMES]

Wow, that looks awesome.

Zooming in, the navigation, with multiple page views.  This really is far superior to the current pdf viewer.

[Vasik Greif]

> Wow, that looks awesome.
> Zooming in, the navigation, with multiple page views.  This really is far
> superior to the current pdf viewer.

Yes!! They have full-text search, voice to text and bunch of other
features too. The way I did it is more a proof-of-concept, it doesn't
have much sense to try to implement the other features before someone
finds a way to specify an array of images as the source for the
book..:-(

[mrpatulski]

@Vaisck

Impressive and quick. Questions:

1. Is this in the basecode or as a plugin?
2. How does RS build the page list now say of a PDF it rasterizes vs the IA ereader?

[Vasik Greif]

> @Vaisck
> Impressive and quick. Questions:
>

>    1. Is this in the basecode or as a plugin?
>    2. How does RS build the page list now say of a PDF it rasterizes vs the
>    IA ereader?

Let me answer both questions by describing what I did:

When PDF gets uploaded, the preview images are created in
preview_preprocessing.php, with random names. I basically duplicated
this code and modified it to store the images into specified folder
inside the resource folder (ie. resourcefolder/book/) and with
sequentional names (001.jpg, 002jpg etc).
Than I created a plugin, which replaces the previewimage hook in
preview.php with the IA reader. I basically took the BookReaderDemo,
copy/pasted the BookReaderJSSimple.js code and just specified the path
to the resourcefolder/book and number of pages. Then the reader
automatically takes the images and builds book.

So, while this works, this is not a good solution, as the preview
images are created twice now. Ideally, we should use the previews that
are already done by RS, and just tell the reader to build a book -
that way we won't screw up anything inside RS, there would be no need
to modify the core, and the reader could be displayed just by simple
plugin.

The function to get the images in JS is (taken from the
BookReaderJSSimple.js):

// We load the images from archive.org -- you can modify this function
to retrieve images
// using a different URL structure
br.getPageURI = function(index, reduce, rotate) {
// reduce and rotate are ignored in this simple implementation,
but we
// could e.g. look at reduce and load images from a different
directory
// or pass the information to an image server
var leafStr = '000';
var imgStr = (index+1).toString();
var re = new RegExp("0{"+imgStr.length+"}$");
var url = 'http://www.archive.org/download/BookReader/img/
page'+leafStr.replace(re, imgStr) + '.jpg';
return url;
}

For sure there's a function in RS to get the array of images previews,
that belong to the pdf (I can check that tomorrow, or Dan can tell
us), so what we need is to modify the JS code to accept the array. I'm
sorry, I don't know anything about JS. Anyone?

[David Dwiggins]

Hi, Vasik and Matt

I just spent a couple of hours working with the IA simple example, and was able to get it to pull the pre-existing images from the RS database. What I did is a really rough proof of concept, but it shows that it can be done.

I didn't do a screencast, but it looks pretty much like what Vasik did only without the RS interface around it yet. 

The trick is to generate the activating javascript and html from php code, so that you can insert the necessary data from the ResourceSpace database. I loaded the list of file URLs for the PDF into a javascript array, and then modified the URL function in the example script to get it to read from the array rather than making up a plain numeric URL.  This could probably also be done on the fly with ajax, although I'm not sure there's much benefit to doing it that way.

This would take some work to get it to production quality, but it's definitely do-able. Considerably more development would be required to support the advanced features of the reader like arbitrary scaling, search within the book, etc. But even with the basic "simple" example it's an improvement over the current built-in pager.

I think it could easily be developed as a plugin or added to the base code. One issue might be figuring out how the open source license is worded and whether distributing it within the RS base is a problem. I doubt it, since GPL and BSD licenses are both pretty permissive. But probably needs to be looked at before it can be put in the base. Putting it a plugin might be less of an issue, since it would be self contained.

Don't have any more time to play with this tonight, but figured I'd give a progress report.

-David

--
You received this message because you are subscribed to the Google Groups "ResourceSpace" group.

[David Mac (new)]

This is really impressive, and so much better than the current
reader / display!

[Vasik Greif]

Now this is cool, thanks David, exactly what was needed to be done!!!

Just to maybe save you some time, here's the code that should work for
getting the array into book plugin (there might be better way, I took
this from resource_functions.php):

<?php
function HookBookAllpreviewimage ()
{
$ref=getval("ref","");
if ($resource['file_extension'] == "pdf") {

// get pages files
$page = 1;
$filearray=array();
$misscount = 0;
// just to be safe, we'll try at least 4 pages ahead to make sure
none got skipped
while($misscount < 4){
$thepath =
get_resource_path(270,true,"scr",false,'jpg',-1,$page,"","","");
if (file_exists($thepath)){
array_push($filearray,$thepath);
$page++;
} else {
$misscount++;
$page++;
}
}
$pagesno=count($filearray);
?>

<!-- code from SimpleBookDemo.js goes here -->

<?php
return true;}
}
?>

For getting the array into JS array, one could do something like:

var myarray = new Array(<?php echo "'".implode("','",
$filearray)."'"; ?> );

or use json_encode function..

Thanks
Vasik

[Vasik Greif]

ups, I missed

$resource=get_resource_data($ref);

in the code...
Vasik

[mrpatulski]

David and Vasik,

Thanks for your enthusiasm. Did you all see this article--may help:

http://raj.blog.archive.org/2011/03/17/how-to-serve-ia-style-books-from-your-own-cluster/

As the non-developer here, I can only keep asking questions:

1. What would it take to develop to a production release plugin ie time/budget?
   
2. Would we be able to embed it elsewhere like we can with video? 
   
3. Would we be able to skin it?
   
4. How would ensure  that it does not get too customized so the code is easy to keep current if IA makes updates?

[Vasik Greif]

uupps, and the line

get_resource_path(270,true,"scr",false,'jpg',-1,$page,"","","");

should obviously be

get_resource_path($ref,true,"scr",false,'jpg',-1,$page,"","","");

[Vasik Greif]

> As the non-developer here, I can only keep asking questions:
>

>    1. What would it take to develop to a production release plugin ie
>    time/budget?
>    2. Would we be able to embed it elsewhere like we can with video?
>    3. Would we be able to skin it?
>    4. How would ensure  that it does not get too customized so the code is

>    easy to keep current if IA makes updates?

1. Developing a basic plugin should be super-easy - the code I posted
above should be enough to display the book, when David uses his
modified BookReaderSimple.js function.
2. Should be easily doable with iframe.
3. The reader uses some images and css styles - not difficult to
modify.
4. In the current state, the only modification to the reader code is
the function that get's the images array - this shouldn't cause any
problems with updates.

The more difficult things are, as David stated above, implementing the
full-text search, text-to-voice, arbitrary zooming etc..

[David Dwiggins]

I've done the basic work on this, and it can be seen in action at

http://rs.dwiggins.net/rstest4/?c=2&k=3434083996

There are still some UI issues and bugs (including a major one where the frameset gets broken if you click the title link at the top.) But it is mostly working. Right now there is little or no change to the underlying IA code, but that might change if we pursue more customization of the look and/or fixing the frameset issues. But I think it should be possible to keep it fairly upgradable.

In order to get it to work properly, I told it to use the maximum dimension of all pages for each page in a document. This doesn't seem quite right, and appears to cause distortion on some pages. (See the Peabody Sisters book, for example.)  But when I tried feeding it the sizes of the individual pages, the turner mechanism didn't seem to work. This would take some additional analysis to figure out. It works tolerably for most documents as-is, though.

Would also need to do some analysis to ensure that this properly respects security restrictions within RS, and would need to spend some more time to look at supporting embedding.

I am into this for about 5 hours of development time so far. It's hard to say, but I'd guess it's another 4-6 hours to get it polished up as a finished plugin.

Let me know if you want to discuss.

-David

--
You received this message because you are subscribed to the Google Groups "ResourceSpace" group.

[Vasik Greif]

Hi David, works great! I'm not able to test the frameset issue,
clicking on the link redicrects me to the login page.
Why is it necessary to specify dimensions? During my test I didn't
fetch any dimension info into the js and it seemed to work well...?

Vasik

[Henrik]

Really nice!

Best regards,
Henrik

[David Dwiggins]

The demo used a fixed page size and all pages were the same size. The size used was close enough to the specs for the ResourceSpace images that it more-or-less worked. However, in cases where the PDF images will vary in size (which is actually the case for the ResourceSpace previews, since their proportions vary) the bookreader ideally requires you to be able to return the width and height of each page so that it can scale them proportionally.

Here's their description of this:

In the case of the Standalone Demo, operation is fairly simple. Images are numbered sequentially and stored in a directory called "StandAloneImages". The images are all the same size, and two functions in BookReaderJSSimple.js, getPageWidth() and getPageHeight(), return the page size. Scaling is done in the web browser.

For books scanned by the Internet Archive and stored on archive.org, bookreader operation is a bit more complex. During the book scanning process, each page is imaged using a high-resolution digital camera, and then each page is cropped and deskewed. The size of the cropped image is stored in a file called scandata.xml. During this process, some images, such as color cards, white cards, and tissue paper pages, are marked as pages that should not be displayed. This information is also stored in scandata.xml. The cropped and deskewed images are stored in JPEG 2000 format in a zip file called bookid_jp2.zip. The raw images, the cropped and deskewed images, and the scandata.xml file are available for each book on archive.org.

Because crop boxes can vary between pages for Internet Archive books, the getPageWidth() and getPageHeight() functions can return a different size for each image in the book. Also, since some pages are not supposed to be displayed, the getPageURI() function maps an "index number" that the book reader uses to a "leaf number" that corresponds to an image in the jp2.zip file. These functions use information from scandata.xml to determine size and url for each page.

However, when I tried implementing this, it didn't seem to work right. So this might require some more testing.

I think the problem would become extremely obvious if you had a PDF with drastically different page shapes. (IE, both vertical and horizontal pages.)

-David

[Tom Gleason]

That's really cool. I wonder if it should go in an iframe so that we
could have a back button and other normal UI.
The full-text search function is also very cool; I wish I had known
about it when I was working on the same feature, but it looks like it
was about the same time. It would be nice too to be able to get
dynamically generated snippet shots of the search results like that.
If we take the pdf search inside functionality any further, we'll
probably try to integrate with this viewer.

Tom

--
Tom Gleason
Vice President, Engineering
Colorhythm LLC
http://www.colorhythm.com

Main Office: +1 415-399-9921
Fax: +1 253-399-9928
Mobile: +1 347-537-8465

tgle...@colorhythm.com

[mrpatulski]

I had dan develop a document viewer embed plugin that uses an iframe--I could pass that code along if you all think it might be of use.

[David Dwiggins]

Yeah, I thought about that, but the iframe-within-a-frame setup might get a bit messy. I think the better solution might be to try to integrate the RS UI features like previous/next, etc. into the top bar of the reader.

Matthew - are you talking about a frame for within the preview screen, or an embeddable iframe that shows the document elsewhere?

[Vasik Greif]

Hi David, any news on this?

Thanks
Vasik

> > > On Thu, Feb 23, 2012 at 3:22 AM, Vasik Greif <vaclav.gr...@gmail.com>

> > tglea...@colorhythm.com

[David Dwiggins]

Hi, Vasik,

Truthfully, haven't had a chance to work on it again due to some other work for clients. I will try to come back to it in the next week or so if I get a bit of time.

-David

[Vasik Greif]

Hi David, just trying to bump this again: have you found time to finish this? 

Thanks

Vasik

Dne úterý, 28. února 2012 0:26:42 UTC+1 David Dwiggins napsal(a):

-David

> > >> resourcespace+unsubscribe@googlegroups.com.

> > >> For more options, visit this group at
> > >>http://groups.google.com/group/resourcespace?hl=en.
>
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "ResourceSpace" group.
> > > To post to this group, send email to resour...@googlegroups.com.
> > > To unsubscribe from this group, send email to

> > > resourcespace+unsubscribe@googlegroups.com.

> > > For more options, visit this group at
> > >http://groups.google.com/group/resourcespace?hl=en.
>
> > --
> > Tom Gleason
> > Vice President, Engineering
> > Colorhythm LLC
> >http://www.colorhythm.com
>
> > Main Office:  +1 415-399-9921
> > Fax: +1 253-399-9928
> > Mobile:  +1 347-537-8465
>

> > tglea...@colorhythm.com

>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "ResourceSpace" group.
> > To post to this group, send email to resour...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > resourcespace+unsubscribe@googlegroups.com.

> > For more options, visit this group at
> >http://groups.google.com/group/resourcespace?hl=en.

--
You received this message because you are subscribed to the Google Groups "ResourceSpace" group.
To post to this group, send email to resour...@googlegroups.com.

To unsubscribe from this group, send email to resourcespace+unsubscribe@googlegroups.com.

[David Dwiggins]

Hi, Vasik,

No, I haven't -- I have been focused on other projects. I committed the code as it was to the plugin repo, so someone else is welcome to pick up on it if they want. I know Tom took a look at it a while back...

-David

To view this discussion on the web visit https://groups.google.com/d/msg/resourcespace/-/9BBOaqBeTv0J.

To post to this group, send email to resour...@googlegroups.com.

To unsubscribe from this group, send email to resourcespac...@googlegroups.com.