Hackathon at ESWC

[Alexander Garcia Castro]

Hi all, we are having a Hackathon while at the ESWC.

May 27 in Montpillier, France.

Theme:



The ability to extract meaningful, machine-interpretable data from

scholarly publications in PDF form is a big challenge. Several open

source libraries exist that attempt to automate this process, but work

needs to be done on them to improve accuracy and reliability. Some

specific and relevant challenges include:



Ability to automatically identify and tokenize citations from the PDF

(or more accurately, from a string of text)

Ability to automatically identify those blocks of text that represent

the narrative in a PDF.

Ability to identify references within the narrative, extract their

scope, and associate them with citation information in the PDF.



Anybody interested is welcome to join us, http://scholrev.org/hackathon/

Please contact Casey McLaughlin <casey.mc...@cci.fsu.edu>

--
Alexander Garcia
http://www.alexandergarcia.name/
http://www.usefilm.com/photographer/75943.html
http://www.linkedin.com/in/alexgarciac

[Leonard Rosenthol]

I would also put forth that hacks related to improving the PDF PRODUCTION
process by standard tooling used by the scholarly community would also go
a LONG WAY to improving the extraction later on.

As mentioned many times, PDF already has facilities for storing VERY RICH
semantics that accompany the presentation. Unfortunately, most open
source tools don't bother to use these features of PDF when creating the
PDF. Additionally, many of them will removing them from the PDF as part
of post-production processes. Fixing the production and post-production
tools to enable the production and processing of semantically-rich PDFs is
extremely important as well.

Leonard Rosenthol | PDF Architect · Principal Scientist | Adobe
Systems Incorporated | leon...@adobe.com



On 4/22/13 2:48 PM, "Alexander Garcia Castro" <alexg...@gmail.com>
wrote:

>--
>You received this message because you are subscribed to the Google Groups
>"Beyond the PDF" group.
>To unsubscribe from this group and stop receiving emails from it, send an
>email to beyond-the-pd...@googlegroups.com.
>For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Cartic Ramakrishnan]

Hello Alexander,

In case you have have not already seen this,

please take a look at http://code.google.com/p/lapdftext/

I did not see this tool listed on your hackathon webpage.

If you think it is relevant and useful please do publicize
this tool too.

Thanks

--
You received this message because you are subscribed to the Google Groups "Beyond the PDF" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beyond-the-pd...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Cartic Ramakrishnan Ph.D.
Research Scientist
Computational Advertising @ DataPop
www.DataPop.com

[Alexander Garcia Castro]

thanks, we will include it. hope u come to the Hackathon

[Phillip Lord]

Leonard

Would adobe be willing to make contributions to the tools that are
commonly used in scientific publication? Given that PDFs produced by
academics are normally generated by with Word or TeX (sometimes
indirectly), it's a fairly limited tool set. Unless these tools are
enhanced, the situation will remain as it is; developing bespoke PDF
production tools won't help, because most of the people who produce PDF
don't want to extract form them.

I don't know what tooling publishers use to produce PDF, nor whether
publisher PDFs are any better in terms of metadata, although I am sure
some one on the list could tell us. Interestingly, though, most
publishers DO put metadata on their webpages, even if it's a little
erratic at time, and they DO put metadata into crossref. I'd be
interested to understand whether the equivalent data is present in the
PDF. Again, I suspect someone could tell us.

Phil

[Leonard Rosenthol]

What sort of contributions are you thinking of, Phil?

We already include, with Adobe Acrobat, tools that integrate directly with
MSWord (for Windows) for high-fidelity conversion of Word->PDF that
include the ability to incorporate rich semantics as part of that
conversion. Additionally we offer an application for mobile devices (iOS
and Android) as well as an online service (createpdf.adobe.com) that do
the same.

In terms of TeX, I am often consulted by the various folks working on PDF
support in the TeX applications on compatibility and/or implementation
issues. It's how/why modern versions of TeX (such as pdfTeX) are able to
produce the same sort of (or sometimes even better!) rich semantic PDFs.

So for these two production systems - I believe the tools are well along.
That's not to say they are perfect, and we'd certainly love to get
feedback on what is missing. (I have my own ideas but they are usually
more useful coming from the actual users :).

Understand that I am not talking about simple metadata, but rich semantics
such as the structure of the content, or even the original source material
from which to derive information. PDF supports MANY different types of
semantic storage, including (but not limited to):

- Embedding the complete source document
- Attaching source fragments to individual graphic objects
(for example, associating the SVG of a vector graphic or MathML of an
equation)
- Attaching data or data sources to graphics
(for example, associating CSV or XML with a graph or chart)
- Applying XML/RDF-based metadata with individual graphic objects or text
- Applying semantic structure to textual content
(paragraphs, tables, columns, callouts, etc.)
- Associating measurement information with 2D or 3D graphics
(enabling real-world coordinate systems including GIS-based ones)

BUT just because PDF supports all of these excellent features - they are
USELESS if the production tooling doesn't support a way to incorporate
them. I know that our Word->PDF converters as well as modern versions of
pdfTeX support some/all of those options listed above. Users, however,
have to choose to use them - both the tools and the options.

Leonard

[Alexander Garcia Castro]

today, do we still need the PDF? how is the PDF interoperable with the
Web of Data? should the PDF be the prominent format for delivering
content? why not XML/RDF and mashups for endusers? why is it that most
scientific articles are so PDF centric?

[Phillip Lord]

Leonard Rosenthol <lros...@adobe.com> writes:
> What sort of contributions are you thinking of, Phil?

Well, you said:

"I would also put forth that hacks related to improving the PDF
PRODUCTION process by standard tooling used by the scholarly community
would also go a LONG WAY to improving the extraction later on."

Mostly, I like to work with web formats, and have never tried to extract
stuff from PDF. So I would hope that you would have a better
understanding of this than I would.

> PDF supports MANY different types of semantic storage, including (but
> not limited to):
>
> - Embedding the complete source document
> - Attaching source fragments to individual graphic objects
> (for example, associating the SVG of a vector graphic or MathML of an
> equation)
> - Attaching data or data sources to graphics
> (for example, associating CSV or XML with a graph or chart)
> - Applying XML/RDF-based metadata with individual graphic objects or text
> - Applying semantic structure to textual content
> (paragraphs, tables, columns, callouts, etc.)
> - Associating measurement information with 2D or 3D graphics
> (enabling real-world coordinate systems including GIS-based ones)
>
> BUT just because PDF supports all of these excellent features - they are
> USELESS if the production tooling doesn't support a way to incorporate
> them. I know that our Word->PDF converters as well as modern versions of
> pdfTeX support some/all of those options listed above. Users, however,
> have to choose to use them - both the tools and the options.

Okay, so then I can see three possible problems. Either, users are not
aware of these semantic options. Now, my poison is latex, so I'd be
happy to understand how to embed (and extract) this sort of semantics;
but I don't know how to. So I would guess that this is the part of the
problem.

The second possible problem is that the tools are too hard to use. Given
that I don't know how to do this with latex, I can't say for sure, but
I'd love to find out.

Finally, the third problem is that not that they are too hard to use per
se, but that the gain and reward for doing so is not worth the effort.

Of the stuff that you talk about, the things that I would find most
interesting are simple bibliographic knowledge (who, what, when and were
was this PDF published), and the semantic structure of documents.

Is there an nice easy start, "how-to-add-this" document? If not, this
would be a good contribution. And a "how-to-get-this-back-out-again". If
not, that would be a better contribution.

Phil

[Leonard Rosenthol]

On 4/23/13 9:42 AM, "Alexander Garcia Castro" <alexg...@gmail.com>
wrote:

>today, do we still need the PDF? how is the PDF interoperable with the
>Web of Data? should the PDF be the prominent format for delivering
>content? why not XML/RDF and mashups for endusers? why is it that most
>scientific articles are so PDF centric?

Excellent questions, AlexanderŠ

I'll give you my opinion, based on talking to numerous individuals,
governments and enterprises around the worldŠbut it's certainly just that
- my own. I am sure others will have their positions and it would be good
to hear them and their justifications.

The main reason why you still need PDF is that it is self-contained,
archival-ready, container for content & metadata.

The web is a wonderful thing - but it is ephemeral and dynamic. Sometimes
that's a great thing, such as when you want to publish your data as it is
being gathered in order to gather input from the community. Sometimes
it's your worst nightmare, such as when your service provider goes out of
business and takes your data with them - or institutes a "pay wall". And
that¹s just today when we think about immediate access. What about the
future? 5, 10, 50, 100 years? I expect that you want to ensure that your
colleagues of the future will be able to learn from you, as you have from
your predecessors. For that, you need something that is self-contained
and archivable - and that's where PDF (and PDF/A) comes in.

I would never suggest that you only publish in PDF and not on the web.
However, there is today and for the future, reasons to also publish in
PDF.

Leonard

[Leonard Rosenthol]

On 4/23/13 9:44 AM, "Phillip Lord" <philli...@newcastle.ac.uk> wrote:
>Mostly, I like to work with web formats, and have never tried to extract
>stuff from PDF. So I would hope that you would have a better
>understanding of this than I would.

Many of the same things can be accomplished with our tools and web formats
such as XML and HTML. However, this is indeed a place where there isn't
support from other places and formats (as there is from LibreOffice,
pdfTeX and even Microsoft). The vast majority of tools that convert
XML/HTML->PDF do so strictly for presentation and don't bother with
semantics - since they see it simply as an alternative to printing (aka a
"take away"), rather than a viable alternative publication format.

We are definitely investing in improving this - both in our own tooling as
well as working with various vendors to help them improve their results.
Unfortunately, we are but a single (loud, but lonely) voice in the wind.
We need users, such as yourselves, to talk to the tool vendors and tell
them how important this stuff is to you.

>Okay, so then I can see three possible problems. Either, users are not
>aware of these semantic options. Now, my poison is latex, so I'd be
>happy to understand how to embed (and extract) this sort of semantics;
>but I don't know how to. So I would guess that this is the part of the
>problem.

Correct, lack of education for users is a huge problem. Myself and others
in the PDF community try to do this as best we can, but there are so many
people that we can't reach.

For LaTeX specifically, I don't use the tools myself so can't comment.
However, Ross Moore is the expert in this area, and he has a series of
videos of his TUG presentations. The most recent is at
<http://river-valley.tv/further-advances-toward-tagged-pdf-for-mathematics/

>.


>The second possible problem is that the tools are too hard to use. Given
>that I don't know how to do this with latex, I can't say for sure, but
>I'd love to find out.

Again, I don't know the specifics for (La)TeX - but I know that they are
trying to make it as simple as possible. In Word, its just a couple of
checkboxes (either with our tools or Microsofts) to choose the options you
wish.

>Finally, the third problem is that not that they are too hard to use per
>se, but that the gain and reward for doing so is not worth the effort.

It's that their use increases file size. Of course, the more (meta)data
you put into something that larger it is going to get. However, many users
still have old-school thoughts of file size concerns OR they are have real
world system issues with quotas of their file servers, mail systems, etc.

>Of the stuff that you talk about, the things that I would find most
>interesting are simple bibliographic knowledge (who, what, when and were
>was this PDF published), and the semantic structure of documents.
>
>Is there an nice easy start, "how-to-add-this" document? If not, this
>would be a good contribution.

There are a lot of great resources out there for "how to add this", both
from Adobe and others. Here are a few that I found on a quick Google
search.

http://webaim.org/techniques/acrobat/converting

http://www.adobe.com/accessibility/products/acrobat/pdf/A9-accessible-pdf-f
rom-word.pdf

http://office.microsoft.com/en-us/word-help/create-accessible-pdfs-HA102478
227.aspx

https://help.libreoffice.org/Common/Export_as_PDF

>And a "how-to-get-this-back-out-again". If
>not, that would be a better contribution.
>

We're working on that! We've worked with the folks at iText to put
together some sample code to show how to do this, and are trying to get
some more examples out thereŠespecially based on open source solutions.

Leonard

[Peter Murray-Rust]

I am at Open Data Workshop in London http://www.w3.org/2013/04/odw/ where Jim King from Adobe has been suggesting very much the same as Leonard. He outlined the various features - ISO standard, attachment of documents, inclusion of metadata, etc.

I agree that it is technically possible to create a moderate degree of semantic interoperability if everyone used conformant PDF tools and also used a common schema for authoring. However this requires a considerable amount of community discipline and also requires purchasing commercial tools. As a result it isn't very attractive to the academic community.

Worse , the PDF produced by almost all STM publishers is seriously non-standard. They are under no market pressure to produce anything better. The production is outsourced to a small number of companies which AFAIK do not contain scientists.  So the prevailing practice is that poor PDF is good enough. Moreover the publishers compete against each other and so even if they used PDF as a container they would all do it differently.

Creating good uniform schema-based PDF is as challenging as creating XML schemas or RDF schemas. I don't see it happening.

And we are very well on the way to creating an Open-source PDF2XML reader... which gives us, rather the publishers the ontological initiative.

P.

--
You received this message because you are subscribed to the Google Groups "Beyond the PDF" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beyond-the-pd...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Peter Murray-Rust
Reader in Molecular Informatics
Unilever Centre, Dep. Of Chemistry
University of Cambridge
CB2 1EW, UK
+44-1223-763069

[Leonard Rosenthol]

Tooling for PDF that supports this, as I mentioned (and I believe Jim did as well) is available NOT JUST for commercial solutions but also in the open source world.  LibreOffice, iText, pdfTeX, Poppler and more are have the pieces that are required.  But as Phil said, they all require user investment.  HOWEVER that investment is going to be required regardless of the format(s) in use.  Even for the "open web", you still need to define those same semantics and schemas and then get tool vendors to support them (even if that's just to avoid screwing them up and/or dropping them on the floor).

What makes you think that the STM publishers are going to do a better job with some other format – eg. XML or HTML – than they do with PDF?

While investing in PDF2XML conversion for LEGACY content is a worthwhile thing, it is very much the WRONG APPROACH going forward – since by the time the content hits PDF, when poor tooling is used, you've lost important semantics.  Sure, you might be able to guess such things as paragraphs or even tables…but equations, no!  Formulas used to produce table values, no!  Measurement information for graphics, no!  (etc.)

But as I said, PDF should not be the only publication format – it should be one format.  And it should be as semantically rich as possible, since it's the format that is going to survive the test of time.

Leonard

[Phillip Lord]

Leonard Rosenthol <lros...@adobe.com> writes:

> On 4/23/13 9:42 AM, "Alexander Garcia Castro" <alexg...@gmail.com>
> wrote:
>
>>today, do we still need the PDF? how is the PDF interoperable with the
>>Web of Data? should the PDF be the prominent format for delivering
>>content? why not XML/RDF and mashups for endusers? why is it that most
>>scientific articles are so PDF centric?

> I'll give you my opinion, based on talking to numerous individuals,
> governments and enterprises around the worldŠbut it's certainly just that
> - my own. I am sure others will have their positions and it would be good
> to hear them and their justifications.
>
> The main reason why you still need PDF is that it is self-contained,
> archival-ready, container for content & metadata.

Although, sadly, as you say, most of the tools that produce PDFs don't
add any metadata.

> The web is a wonderful thing - but it is ephemeral and dynamic.

Just as true of PDFs on the web, as HTML.

> Sometimes that's a great thing, such as when you want to publish your
> data as it is being gathered in order to gather input from the
> community. Sometimes it's your worst nightmare, such as when your
> service provider goes out of business and takes your data with them -
> or institutes a "pay wall". And that¹s just today when we think about
> immediate access. What about the future? 5, 10, 50, 100 years?

Steve Pettifer describes PDF as the elephant in the room. He's right for
scientific publishing. But HTML is the blue whale in the room (broken
metaphor) at large. For every document in PDF, I would guess that there
are 100s on the web.

In 100 years time, will tools still read the PDF? Or will they read the
most common document format around? That's HTML.

(Actually, probably that's still paper, but you know what I mean...).

> I expect that you want to ensure that your colleagues of the future
> will be able to learn from you, as you have from your predecessors.
> For that, you need something that is self-contained and archivable -
> and that's where PDF (and PDF/A) comes in.

HTML with data-uri's are also self-contained and archivable. Or a WARC.

> I would never suggest that you only publish in PDF and not on the web.
> However, there is today and for the future, reasons to also publish in
> PDF.

I agree, although my reasons are more prosaic. PDFs are good in the
bath. HTML doesn't look as nice on paper.

Phil

[Peter Murray-Rust]

On Tue, Apr 23, 2013 at 4:56 PM, Leonard Rosenthol <lros...@adobe.com> wrote:

 

What makes you think that the STM publishers are going to do a better job with some other format – eg. XML or HTML – than they do with PDF?

I don't think that. I am simply saying that creating good **community** PDF standards is hard and unlikely to happen.

While investing in PDF2XML conversion for LEGACY content is a worthwhile thing, it is very much the WRONG APPROACH going forward – since by the time the content hits PDF, when poor tooling is used, you've lost important semantics.

yes - but it is currently the ONLY way forward.

 

 Sure, you might be able to guess such things as paragraphs or even tables…but equations, no!

We can extract chemical equations and formulae from PDFs. I also expect to make some progress with maths.
 

 Formulas used to produce table values, no!  Measurement information for graphics, no!  (etc.)

Phylogenetic trees. Yes. Not 100% - perhaps 25%.

But as I said, PDF should not be the only publication format – it should be one format.  And it should be as semantically rich as possible, since it's the format that is going to survive the test of time.

Maybe. maybe HTML5? maybe EEG traces? who knows.
 

[Jodi Schneider]

Hi Leonard,

I think we're on the same page, mostly. But we're not publishers, we're scientists trying to get what we need out of publishers' PDFs (ironically created by ourselves and our colleagues). And this is the problem.

On Tue, Apr 23, 2013 at 4:56 PM, Leonard Rosenthol <lros...@adobe.com> wrote:

While investing in PDF2XML conversion for LEGACY content is a worthwhile thing, it is very much the WRONG APPROACH going forward – since by the time the content hits PDF, when poor tooling is used, you've lost important semantics.  Sure, you might be able to guess such things as paragraphs or even tables…but equations, no!  Formulas used to produce table values, no!  Measurement information for graphics, no!  (etc.)

We very much agree! 

But this is the world of STM publishing: produce the PDF and sell access to it (to libraries, individual readers, or authors willing to pay open access fees). Markup produced is an ASSET for the publisher -- not to be shared. Not even with the author who produced the intellectual content.

This is why we as individual scientists see PDF2XML processes as the way forward for accessing content -- and why some of us on this thread have written papers and code [1] to move the state-of-the-art of PDF extraction forward.

But as I said, PDF should not be the only publication format – it should be one format.  And it should be as semantically rich as possible, since it's the format that is going to survive the test of time.

I heartily agree, and I've been microblogging about it with every time I see another format (whether HTML or ePUB) newly pushed out by a publisher. But in STM academic publishing this is still rare.

We need a standard for publishing STM PDF -- just as PDF/A is a standard for digital preservation.

Under an PDF/STM standard, semantic rich markup would be embedded *within* the PDF. 

This goes back to Gully Burns' comments at the first Beyond the PDF conference: "this is an example of the kind of thing that common practices lead to... the supplemental material of this paper I cited was a 40 page PDF... the page itself is represented here, the font was so small I couldn't read it. The data itself... The author wanted to put his data within the publication, the only mechanism he was provided for that was putting a PDF... there was no thought about trying to make this available for reuse."

Are there any current initiatives about a data-friendly standard for PDF, for instance? Or embedding semantics and full-text within a PDF?

-Jodi

[1] e.g. the aforementioned https://code.google.com/p/lapdftext/

[2] http://www.youtube.com/watch?v=nSUzmKNXufE&list=PLBE627F48A0DB94FD&t=9m17s

[Maria Liakata]

Hello all

I wanted to make people aware of a shared task for combining different types of discourse information in scientific articles, at the LAW VII-D workshop at ACL 2013

http://www.linguistics.ruhr-uni-bochum.de/law7-id/
http://nactem.ac.uk/law7-id/sharedtask.html

The scientific texts available here have already been converted from XML to a standoff format, which permits visualisation. Unlike the hackathon, this workshop does not address the issue of conversion from pdf, but can give you some extra ideas about type of information to extract, other than citations.

Btw, a student of mine has had reasonable success with combining pdfx (http://pdfx.cs.man.ac.uk/) with a system for scientific discourse annotations which assumes XML input (SAPIENTA http://www.sapientaproject.com/software) in the Partridge project (http://papro.org.uk/2013/02/05/partridge-now-open-for-public-testing/).

Best

Maria

--
Dr Maria Liakata
Assistant Professor
Department of Computer Science
University of Warwick
Tel: +44 (0) 24 7652 3681
http://www2.warwick.ac.uk/fac/sci/dcs/people/maria_liakata
--------
Visiting fellow
Text mining group
EBI-EMBL
Tel: +44 (0) 1223 494577

On Mon, Apr 22, 2013 at 7:48 PM, Alexander Garcia Castro <alexg...@gmail.com> wrote:

Hi all, we are having a Hackathon while at the ESWC.

May 27 in Montpillier, France.

Theme:



The ability to extract meaningful, machine-interpretable data from

scholarly publications in PDF form is a big challenge.  Several open

source libraries exist that attempt to automate this process, but work

needs to be done on them to improve accuracy and reliability.  Some

specific and relevant  challenges include:



Ability to automatically identify and tokenize citations from the PDF

(or more accurately, from a string of text)

Ability to automatically identify those blocks of text that represent

the narrative in a PDF.

Ability to identify references within the narrative, extract their

scope, and associate them with citation information in the PDF.



Anybody interested is welcome to join us, http://scholrev.org/hackathon/

Please contact Casey McLaughlin <casey.mc...@cci.fsu.edu>

--
Alexander Garcia
http://www.alexandergarcia.name/
http://www.usefilm.com/photographer/75943.html
http://www.linkedin.com/in/alexgarciac

[Leonard Rosenthol]

The recently published PDF/A-3a is going to be the best route for semantically & data-rich publications. In fact, we specifically added some new features to PDF and PDF/A just for those purposes based on the feedback of this group and others!!

Of course, having just been published within the last 6 or so months, the tools aren't there yet.   But at least now there is an excellent open standard on which to "hang your hat" and build from.

Leonard

[Leonard Rosenthol]

On 4/23/13 12:07 PM, "Phillip Lord" <philli...@newcastle.ac.uk> wrote:
>Steve Pettifer describes PDF as the elephant in the room. He's right for
>scientific publishing. But HTML is the blue whale in the room (broken
>metaphor) at large. For every document in PDF, I would guess that there
>are 100s on the web.

I think it is quite fair to say that there are many more web pages than
PDF pages in existence. However, HTML pages aren't documents and they
necessarily represent content - they may be applications (eg. Gmail) or
dynamically generated things (eg. Wikipedia). One thing we do know,
however, is that there are orders of magnitude MORE PDFs inside of private
repositories than on the public web - which is exactly the opposite for
HTML.

>In 100 years time, will tools still read the PDF?

I think I can safely say, YES! It's an stable, open international
standard and there are TENS OF BILLIONS of files in that format that will
need to be maintained and potentially readŠ

>Or will they read the most common document format around? That's HTML.

[I don't want to start a religious war, but these distinctions are
important]

First and foremost, HTML isn't a document format. It's a way to present
content in a web browser. This is well described not only by the HTML
specification itself but also by numerous other standards organizations,
including the IDPF who believe that EPUB is the "document format for
HTML-based content". Also, you will find that those standards that
specifically work with documents, such as in the area of archiving,
electronic signatures, etc. define it - they very clearly stipulate that
HTML need not apply.

Second, HTML is a (extremely) moving target! HTML 5, which is still not
complete(!!), is NOT compatible with HTML 4. Not only that there are
numerous things added, BUT there are numerous things removed and/or
changed. There is no expectation that an HTML 4 "user agent" (UA) will
handle HTML 5 - OR that an HTML 5 UA will handle HTML 4Šand even if it
does, perhaps not in the same way that a pure HTML 4 UA would have.

So in 100 years, PDF and PDF/A will still be around and stableŠbut who
knows what HTML will be like and whether it will be possible to read such
content. OR for that matter, if such content will have been maintained
and in what manner?!?

>HTML with data-uri's are also self-contained and archivable. Or a WARC.

Yes, for those parts of the HTML specification that support it. Not all
things do...Also, as you note with issues around PDF tooling - finding
HTML tooling that supports data URIs is quite limited/rare.

WARC or the myriad of other packaging formats and specifications for
creating an "archive" or "container" for HTML-based content are certainly
good ideas. However, there isn't a single standard - there are MANY and
they aren't (necessarily) compatible with each other. So while you might
be able to pack things up, there is no guarantee it can be read in the
future (even tomorrow).


Leonard

[Alexander Garcia Castro]

why do we want to keep the PDF? I mean, what is the advantage in
keeping the PDF vs moving in to a technology that is less thought for
printed paper and more thought for the web? After all, word processors
mostly come from the type writing machine translated to the PC
metaphor but now we are far from either one of them. So again, why
should we keep the PDF? just because it preserves a snapshot of a
document at a given time? cant we do better? just because a lot of
people use the PDF and there are lots of legacy PDFs? there were lots
of CD roms and now days one hardly sees CD ROMS. So, is it that we
cant do any better?

[Leonard Rosenthol]

> why do we want to keep the PDF?
>I mean, what is the advantage in keeping the PDF vs moving in to a technology that is less thought for printed paper and more thought for the web?
>

And what is that technology of which you speak?

If there existing a technology today that offered you the benefits of PDF _AND_ and the benefits of the web - then I am sure that the world (and not just the STM community) would be flocking to it. However, no such thing currently exists (for a variety of reasons). Are there numerous projects and people working to find this happy place? Of course. But we're simply not there yet and won't be for the foreseeable future. AND even if/when we get there, it will most definitely NOT create a forced migration of the billions of documents out there.

>So again, why should we keep the PDF? just because it preserves a snapshot of a document at a given time? cant we do better?
>

As with any other endeavor - it depends entirely on what you define as "better"? or by what criteria you judge "better".

>there were lots of CD roms and now days one hardly sees CD ROMS. So, is it that we cant do any better?
>

Hardware migration is lossless (in almost all cases) but software/format migration is not.

Leonard

>> worthwhile thing, it is very much the WRONG APPROACH going forward -

>> since by the time the content hits PDF, when poor tooling is used,
>> you've lost important semantics. Sure, you might be able to guess

>> such things as paragraphs or even tables...but equations, no! Formulas used to produce table values, no!

>> Measurement information for graphics, no! (etc.)
>
>
> We very much agree!
>
> But this is the world of STM publishing: produce the PDF and sell
> access to it (to libraries, individual readers, or authors willing to
> pay open access fees). Markup produced is an ASSET for the publisher -- not to be shared.
> Not even with the author who produced the intellectual content.
>
> This is why we as individual scientists see PDF2XML processes as the
> way forward for accessing content -- and why some of us on this thread
> have written papers and code [1] to move the state-of-the-art of PDF
> extraction forward.
>
>>

>> But as I said, PDF should not be the only publication format - it

[Phillip Lord]

Leonard Rosenthol <lros...@adobe.com> writes:

> On 4/23/13 12:07 PM, "Phillip Lord" <philli...@newcastle.ac.uk> wrote:
>>In 100 years time, will tools still read the PDF?
>
> I think I can safely say, YES! It's an stable, open international
> standard and there are TENS OF BILLIONS of files in that format that will
> need to be maintained and potentially readŠ

There are lot of LPs still around, but it's getting very hard to buy a
stylus.

>>Or will they read the most common document format around? That's HTML.
>
> [I don't want to start a religious war, but these distinctions are
> important]
>
> First and foremost, HTML isn't a document format. It's a way to present
> content in a web browser. This is well described not only by the HTML
> specification itself but also by numerous other standards organizations,
> including the IDPF who believe that EPUB is the "document format for
> HTML-based content". Also, you will find that those standards that
> specifically work with documents, such as in the area of archiving,
> electronic signatures, etc. define it - they very clearly stipulate that
> HTML need not apply.

Yes, indeed, this is true. Many libraries will not archive HTML. A
mistake, in my book, as it stems from an archaic view that content is
the same thing as a document. Libraries should be archiving the former,
the latter is just a passing phase, I think.

> Second, HTML is a (extremely) moving target!

Well, you just said that PDF/A had just been released.

>
>>HTML with data-uri's are also self-contained and archivable. Or a WARC.
>
> Yes, for those parts of the HTML specification that support it. Not all
> things do...Also, as you note with issues around PDF tooling - finding
> HTML tooling that supports data URIs is quite limited/rare.

Firefox? IE?

> WARC or the myriad of other packaging formats and specifications for
> creating an "archive" or "container" for HTML-based content are certainly
> good ideas. However, there isn't a single standard - there are MANY and
> they aren't (necessarily) compatible with each other.

PDF? PS? And that's just the document standards produced by one company.
As you say, in most cases they are not the *real* document which are
actually .doc, .docx or whatever, with the PDF being the bit between the
tool and the printer.

> So while you might be able to pack things up, there is no guarantee it
> can be read in the future (even tomorrow).

It's about probabilities not guarantees.

Phil

[Peter Murray-Rust]

We have had the same discussion at http://www.w3.org/2013/04/odw/ from Jim King (slides http://www.w3.org/2013/04/odw/Role_of_PDF_and_Opendata_final.pdf).  JK's argument is the because PDF can be used as a container then we should use it to publish data.

It may be useful to summarise the arguments for PDF and then challenge them
.

PDF exists, is widely used, and there is a wide range of commercial tools

PDF *can* act as a container for other objects
PDF *can* contain structured information

PDF *can* contain metadata.

PDF is now an ISO standard

["*can*" means that it has the theoretical possibility of doing this, not that it is used in practice]

On the negative side:

PDF does not, by default support running text, tables, figures or other structured objects.

Most of the other features of PDF argued by JK and LR can be provided by other commercial tools such as Word or Powerpoint (their layout is poorer quality but their treatment of words and characters and tables and document structure is far superior)

It is difficult to create interoperability except by buying commercial tools , especially from Adobe.

It is difficult to innovate by creating derivative works (tools, content, standard, etc.)

Adobe has not been a community player. The PDF standard was developed proprietarily and only Opened after 15 years. In similar vein Adobe supported W3C/SVG initially and then withdrew all support for SVG and the tools it had created. (Note: there is nothing wrong with a company following its commercial

=======

Many toolsets have the potential to transport semantic structured information but few are actually used. All of the features of PDF can be provided by a combination of W3C tools (XML, metadata, graphics, tables, layout (XSL-FO)) and if this were adopted fully with good tools it would represent an open  solution that could be further developed. Similar things can be said for HTML5 and these can represent acceptable architectures.

I have now heard Adobe presenting the virtues of PDF for both structured documents and data and neither has yet convinced me.

--
You received this message because you are subscribed to the Google Groups "Beyond the PDF" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beyond-the-pd...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Alexander Garcia Castro]

I agree with Peter and Phillip. I would just add that I would love to
have someone from ADOBE making the case for the PDF within the context
of the semantic web and the web of data. I am all for it, and
Sepublica (http://sepublica.mywikipaper.org/) could be the place.

[Leonard Rosenthol]

On 4/24/13 7:41 AM, "Phillip Lord" <philli...@newcastle.ac.uk> wrote:
>>Second, HTML is a (extremely) moving target!
>
>Well, you just said that PDF/A had just been released.

PDF/A-3 was just released.

PDF/A-1, however, was released in 2005 and has since been adopted as a
national standard (aka required to be used!) by over 50 countries around
the world (on 5 of 7 continents) and many states in the US.

>>>HTML with data-uri's are also self-contained and archivable. Or a WARC.
>>
>> Yes, for those parts of the HTML specification that support it. Not all
>> things do...Also, as you note with issues around PDF tooling - finding
>> HTML tooling that supports data URIs is quite limited/rare.
>
>Firefox? IE?

Sorry, I meant authoring tools.

>>So while you might be able to pack things up, there is no guarantee it
>> can be read in the future (even tomorrow).
>
>It's about probabilities not guarantees.
>

For the average user, such as you and me, I agree. However when you talk
to archivists and government agencies, there is no such thing as "maybe"Š

Leonard

[Leonard Rosenthol]

Peter – if you don't want to use PDF, that's fine.  It's your content and you should publish it in the format(s) that you want.

My only point throughout this conversation, and that of Jim as well, is that many people dismiss PDF as a format because of the poor tooling that is out there, RATHER than on true technical merit (or lack thereof).   And that as the STM community continues their work towards finding a replacement for PDF – that they shouldn't just throw it out based on the past, but consider how it might fit into the future (if at all).

Leonard

From: Peter Murray-Rust <pm...@cam.ac.uk>
Reply-To: "beyond-...@googlegroups.com" <beyond-...@googlegroups.com>
Date: Wednesday, April 24, 2013 9:31 AM
To: "beyond-...@googlegroups.com" <beyond-...@googlegroups.com>

[Eduard Hovy]

Hi Alexander, 

On Apr 23, 2013, at 6:22 PM, Alexander Garcia Castro <alexg...@gmail.com> wrote:

why do we want to keep the PDF? I mean, what is the advantage in
keeping the PDF vs moving in to a technology that is less thought for
printed paper and more thought for the web? After all, word processors
mostly come from the type writing machine translated to the PC
metaphor but now we are far from either one of them. So again, why
should we keep the PDF? just because it preserves a snapshot of a
document at a given time? cant we do better? just because a lot of
people use the PDF and there are lots of legacy PDFs? there were lots
of CD roms and now days one hardly sees CD ROMS. So, is it that we
cant do any better?

What would you suggest we replace pdf with?  (Of course I don't care whether the underlying implementation of the fonts and presentation formats etc. is pdf or something else, as long as it supports the functionality.  So let's just talk functionality.)

Text 

Whatever replaces pdf and similar still has to be able to display text, with appropriate formatting.  Current shortcomings are the static-ness of the text and the formatting -- I can't easily request a summary of the text ("just give me the 200-word summary as it pertains to my current interest") or change the formatting ("please highlight every sentence and figure that discusses chemical X").  So there's one extension.  But it is probably not that far away from being available; I have seen prototypes of both.  

Hyperlinks 

Already there. 

Figures 

Of course one wants dynamic figures, video, etc.  But this to some degree exists already within pdf.  I have seen pdf documents with movable figures -- graphs that rotate, diagrams in which you can click to remove bits to uncover things below, etc.    

Hyper- composition and decomposition 

It would be nice to have documents 'assemble themselves' given my current interest.  A sort of super-google that both finds the relevant material and extracts the relevant passages and weaves them together.  Modern multi-doc text summarization systems can do the first two steps but are still rather poor at the third.  But this is also not that terribly far off.  

Data 

Sometimes I want mostly data, accompanied by descriptive text and/or figures.  pdf is not so good for this.  Should we add spreadsheet capability into standard pdf packages?  

What else would you do to go beyond pdf? 

E

[Phillip Lord]

Eduard Hovy <eh...@andrew.cmu.edu> writes:
> Data
> Sometimes I want mostly data, accompanied by descriptive text and/or figures.
> pdf is not so good for this. Should we add spreadsheet capability into
> standard pdf packages?

I have my doubts about this. Stuffing everything into one file format
doesn't seem like the best way to go. I mean, am I to stick terabytes of
sequence data into a PDF?

PDF is currently and best used to represent scientific papers. Data
should be XML, CSV, HDF whatever. Conflating everything makes little
sense to me.

Phil

[Fenner...@mh-hannover.de]

Dear Beyond the PDF list,

I am happy to see that the Amsterdam Manifesto on Data Citation Principles is one of the winners of the 1K challenge. This is a good opportunity to announce a new service that should facilitate data citation:

http://datacite.labs.orcid-eu.org

The service allows users to search the DataCite Metadata Store, and to add their research outputs – including datasets, software, and others – to their ORCID profile. This should increase the visibility of these research data, and will make it easier to use these data citations in applications that connect to the ORCID Registry - ImpactStory is one of several services that is already doing that. In addition, the service is also providing formatted citations in several popular citation styles, supports COinS, links to related resources, and displays the attached Creative Commons license where this information is available. The DataCite Metadata Store of course also contains many text documents from academic publishers and services such as figshare or PeerJ Preprints, and these works can also be claimed.

This tool is a collaborative effort by Gudmundur Thorisson (ORCID) and myself with major input by Karl Ward (CrossRef) and Sebastian Peters (DataCite). The source code is a fork of the CrossRef Metadata Search source code written by Karl Ward and is available at http://github.com/mfenner/cr-search. The service is currently in beta, so please expect minor bugs and user interface glitches. The official launch will be at the joint Dryad/ORCID Meeting May 23 in Oxford.

Kind regards,

Martin=

[Alexander Garcia Castro]

Hi Ed, my reply within the text of your original email.

I am not so sure about this. Extracting quality text from the PDF is
not trivial, usually u loose something. for instance, in our specific
scenario we need to extract all the bibliographic references from the
PDF. this taks is not always possible, the quality of the extracted
text varies. Another example, extracting tables and figures is less
problematic. The cases u describe are more the sort of cases currently
being supported by utopia; against some PDFs, it does not work for
every PDF. I would like the text to be free from the PDF. if the PDF
is a format I should be able to easily get the text with no errors in
the extraction. consider the case of having a digital library of over
10000 documents, all PDFs. of various sorts (heterogeneity in layouts,
provenance, content). you cant just index the content in the PDF. it
is not as if u really had text. sure, the format and ability to
display is in the PDF, but content is king, once u have the content
you can focus on what to do with it. We have tested the tools
described at http://scholrev.org/hackathon/ and results are not
exactly great, a lot of manual work is required if u want to have 100%
accuracy. If u want I could share with you some of our datasets (PDFs
and corresponding XML outputs).

>
> Hyperlinks
> Already there.
>
> Figures
> Of course one wants dynamic figures, video, etc. But this to some degree
> exists already within pdf. I have seen pdf documents with movable figures
> -- graphs that rotate, diagrams in which you can click to remove bits to
> uncover things below, etc.

Why not being able to manipulate content in whatever way u want?
figures and tables are in the PDF as static locked content. what if u
would like to have a pie instead of an XY figure for a given table?
why should I have to do hard work in order to extract the information
from tables (this is possible, but still it becomes one more thing to
do). My point is simple, it is true, the PDF is great for human
interoperability. But at the age of e-science we ALSO need machine
interoperability, machine procesable documents -IMHO.

>
> Hyper- composition and decomposition
> It would be nice to have documents 'assemble themselves' given my current
> interest. A sort of super-google that both finds the relevant material and
> extracts the relevant passages and weaves them together. Modern multi-doc
> text summarization systems can do the first two steps but are still rather
> poor at the third. But this is also not that terribly far off.

I am interested in this topic, could u share some papers/docs/URLs about this?

>
> Data
> Sometimes I want mostly data, accompanied by descriptive text and/or
> figures. pdf is not so good for this. Should we add spreadsheet capability
> into standard pdf packages?

I dont just need spreadsheets, I need experimental protocols that can
be followed by humans just as by machines; these experimental
protocols are all locked in PDFs. Even if we had, which I argue we
dont yet have at this time, a feasible human and machine
representation of experimental protocols (EXACT, OBI, etc have not yet
proven to do what I describe), such information would remain locked in
the PDF, useless for machines to use and interpret. Also, consider
genomic information, not something you would like to have just in the
document but something for which u need some other solution. Data,
sure I also want data, but I would like to have a container that is
flexible and open so that it is human readable just as it is machine
prosesable.

>
> What else would you do to go beyond pdf?

understand the paper as an interface to the web of data; understand
the paper as a container, as a flexible open grid you may use to
accomodate research objects represented and structured in ways so that
humans and machines may interpret them. understand how the narrative,
essencial in human communication, could accomodate layers of machine
interoperability. Dont get me wrong, I like the PDF, it looks nice and
when printed it makes a nice document. But, I dont think that an
artifact that was conceptualized and developed for print-paper-based
communication is what we need to support scholarly communication.
consider annotations in DOMEO, how could u sync those against their
corresponding PDFs? also, statistics form PDFs are somewhat poor, how
many times has a PDF been shared over email. printed? how could u know
all the tweets about a paper? just bu using the DOI? what if I dont
report the DOI in the tweet? why is it that I can track banners in
e-business in a very accurate and comprehensive manner but I just cant
when it comes to PDF files in scholarly communication? Anyway, as I
said before, I dint have anything against the PDF, but I think it is
not sufficient as the main artifact upon which we are supporting an
important component of scholarly communication.

> E

[Eduard Hovy]

Hi Alexander, 

Thank you for your comments!  I find these very helpful in growing my own model of what the shortcomings of PDF (and not only PDF) are.  I think your comments can serve as a set of challenge problems for developers.  

E

[Kaveh Bazargan]

As a latecomer to this thread, I am in general agreement with Leonard. I like PDF because it is self contained or, as expressed beautifully by Steve Pettifer (@srp) it has "edges". But like Leonard, I think that putting effort into extracting re-usable content from PDF is not the right approach, going forward, even though PDF has the facility to embed almost any content within it, including complete TeX files, XML, MathML, video, and interactive 3D content. 

In my view the definitive, archivable format of a document should be XML, which has no "form" but only content.  XML was designed to hold structured content, whereas PDF was designed (some 25 years ago) to replicate a printed page. I can't see the advantage of stuffing information into PDF and pulling it out again. 

As a slight digression, the quality of XML created by publishers is varied. Some are good, some close to useless. (Most don't publish the XML, so we can't tell of course.) So I feel our aim should be to push publishers to produce accurate, "pure" XML, and to publish them.

[Dr David Shotton]

Dear folks,

This is an excellent example of 'joined-up' thinking - and it works!
Well done.

David

On 24/04/2013 19:49, Fenner...@MH-Hannover.de wrote:
> Dear Beyond the PDF list,
>
> I am happy to see that the Amsterdam Manifesto on Data Citation Principles is one of the winners of the 1K challenge. This is a good opportunity to announce a new service that should facilitate data citation:
>
> http://datacite.labs.orcid-eu.org
>

> The service allows users to search the DataCite Metadata Store, and to add their research outputs � including datasets, software, and others � to their ORCID profile. This should increase the visibility of these research data, and will make it easier to use these data citations in applications that connect to the ORCID Registry - ImpactStory is one of several services that is already doing that. In addition, the service is also providing formatted citations in several popular citation styles, supports COinS, links to related resources, and displays the attached Creative Commons license where this information is available. The DataCite Metadata Store of course also contains many text documents from academic publishers and services such as figshare or PeerJ Preprints, and these works can also be claimed.

>
> This tool is a collaborative effort by Gudmundur Thorisson (ORCID) and myself with major input by Karl Ward (CrossRef) and Sebastian Peters (DataCite). The source code is a fork of the CrossRef Metadata Search source code written by Karl Ward and is available at http://github.com/mfenner/cr-search. The service is currently in beta, so please expect minor bugs and user interface glitches. The official launch will be at the joint Dryad/ORCID Meeting May 23 in Oxford.
>
> Kind regards,
>
> Martin=
>

--

Dr David Shotton
Research Data Management and Semantic Publishing Research Group
Department of Zoology, University of Oxford
South Parks Road, Oxford OX1 3PS, UK.
Phone: +44-(0)1865-271193 Skype: davidshotton

[Martin Fenner]

Thank you David. There is a lot that can be done to further improve this tool, and the related CrossRef Metadata Search. We are always looking for input via email, Github Issue tracker (https://github.com/mfenner/cr-search/issues), or in person, e.g. at the Dryad/ORCID Meetings in Oxford next week.

Best,

Martin

Am 15.05.2013 um 10:11 schrieb Dr David Shotton <david....@zoo.ox.ac.uk>:

> Dear folks,
>
> This is an excellent example of 'joined-up' thinking - and it works! Well done.
>
> David
>
>
> On 24/04/2013 19:49, Fenner...@MH-Hannover.de wrote:
>> Dear Beyond the PDF list,
>>
>> I am happy to see that the Amsterdam Manifesto on Data Citation Principles is one of the winners of the 1K challenge. This is a good opportunity to announce a new service that should facilitate data citation:
>>
>> http://datacite.labs.orcid-eu.org
>>

>> The service allows users to search the DataCite Metadata Store, and to add their research outputs – including datasets, software, and others – to their ORCID profile. This should increase the visibility of these research data, and will make it easier to use these data citations in applications that connect to the ORCID Registry - ImpactStory is one of several services that is already doing that. In addition, the service is also providing formatted citations in several popular citation styles, supports COinS, links to related resources, and displays the attached Creative Commons license where this information is available. The DataCite Metadata Store of course also contains many text documents from academic publishers and services such as figshare or PeerJ Preprints, and these works can also be claimed.

[Ratner, Howard]

Great stuff! I am really happy to see so many of the identifier players interoperate so nicely.

-- Howard

On May 15, 2013, at 5:11 AM, "Dr David Shotton" <david....@zoo.ox.ac.uk> wrote:

Dear folks,

This is an excellent example of 'joined-up' thinking - and it works! Well done.

David


On 24/04/2013 19:49, Fenner...@MH-Hannover.de wrote:
> Dear Beyond the PDF list,
>
> I am happy to see that the Amsterdam Manifesto on Data Citation Principles is one of the winners of the 1K challenge. This is a good opportunity to announce a new service that should facilitate data citation:
>
> http://datacite.labs.orcid-eu.org
>

> The service allows users to search the DataCite Metadata Store, and to add their research outputs – including datasets, software, and others – to their ORCID profile. This should increase the visibility of these research data, and will make it easier to use these data citations in applications that connect to the ORCID Registry - ImpactStory is one of several services that is already doing that. In addition, the service is also providing formatted citations in several popular citation styles, supports COinS, links to related resources, and displays the attached Creative Commons license where this information is available. The DataCite Metadata Store of course also contains many text documents from academic publishers and services such as figshare or PeerJ Preprints, and these works can also be claimed.

>
> This tool is a collaborative effort by Gudmundur Thorisson (ORCID) and myself with major input by Karl Ward (CrossRef) and Sebastian Peters (DataCite). The source code is a fork of the CrossRef Metadata Search source code written by Karl Ward and is available at http://github.com/mfenner/cr-search. The service is currently in beta, so please expect minor bugs and user interface glitches. The official launch will be at the joint Dryad/ORCID Meeting May 23 in Oxford.
>
> Kind regards,
>
> Martin=

--

Dr David Shotton
Research Data Management and Semantic Publishing Research Group
Department of Zoology, University of Oxford
South Parks Road, Oxford OX1 3PS, UK.
Phone: +44-(0)1865-271193 Skype: davidshotton