TIFF vs. JPEG2000
Creighton Barrett
I'm curious to hear whether you are using TIFF or JPEG2000 as a preservation scan in your digitization workflows. If you switched to JPEG2000, could you shed some light on the reasoning? What has your experience been? If you considered switching but stuck with TIFF, could you explain why? Did you perform tests on image quality, storage requirements, etc.?
Any info would be much appreciated!
Cheers,
Creighton Barrett
Dalhousie University Archives
Christie Peterson
--
You received this message because you are subscribed to the Google Groups "Digital Curation" group.
To unsubscribe from this group and stop receiving emails from it, send an email to digital-curati...@googlegroups.com.
To post to this group, send email to digital-...@googlegroups.com.
Visit this group at http://groups.google.com/group/digital-curation?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
Creighton Barrett
Cliff, Peter
Perhaps a little old now, but you may find useful stuff on the Open Planets Foundation JP2 Working Group (now disbanded I believe):
http://wiki.opf-labs.org/display/JP2/Home
and also the Wellcome Library's blog:
http://www.jpeg2000wellcomelibrary.blogspot.co.uk/
again, last update 2011 but still has interesting things in it.
At the BL we have migrated TIFFs to JPEG2000s and I understand the reason for doing this is that the file sizes are significantly smaller but image degradation is not so bad.
As with all these things it will probably come down to your own use case - you have a load of things that can render TIFF? Keep the TIFF. You have a load of TIFFs and no significant processing power to convert to JP2 even if you wanted to? Keep the TIFF. You want to save disk space? Use JP2.
My feeling is that neither format is a particular preservation risk at the moment, but as Chris' post points out, currently support for JP2 is patchy at best - so we as a community either need to get involved with the tools, be prepared to pay or use something else!
I'm not aware of any efforts to introduce JP2 support to any of the major browsers.
--
Pete Cliff
Digital Preservation Technical Lead
The British Library
-----Original Message-----
From: digital-...@googlegroups.com on behalf of Creighton Barrett
Sent: Mon 18/03/2013 18:35
To: digital-...@googlegroups.com
Subject: Re: [digital-curation] TIFF vs. JPEG2000
Thanks Christie, much appreciated. I didn't realize use of JPEG2000 was still so low. I would love to hear from people who are using it. Is anyone aware of efforts to introduce browser-support?
On 18 March 2013 14:50, Christie Peterson <christie....@gmail.com> wrote:
A recent blog post on the Library of Congress's Digital Preservation blog did the best job I've seen of laying out issues related to JPEG2000 and preservation (read the discussion in the comments, too): http://blogs.loc.gov/digitalpreservation/2013/01/is-jpeg-2000-a-preservation-risk/
Best,
Christie Peterson
On Mon, Mar 18, 2013 at 1:04 PM, Creighton Barrett <csba...@gmail.com> wrote:
Hi everyone,
I'm curious to hear whether you are using TIFF or JPEG2000 as a preservation scan in your digitization workflows. If you switched to JPEG2000, could you shed some light on the reasoning? What has your experience been? If you considered switching but stuck with TIFF, could you explain why? Did you perform tests on image quality, storage requirements, etc.?
Any info would be much appreciated!
Cheers,
Creighton Barrett
Dalhousie University Archives
--
You received this message because you are subscribed to the Google Groups "Digital Curation" group.
To unsubscribe from this group and stop receiving emails from it, send an email to digital-curati...@googlegroups.com <mailto:digital-curation%2Bunsu...@googlegroups.com> .
To post to this group, send email to digital-...@googlegroups.com.
Visit this group at http://groups.google.com/group/digital-curation?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "Digital Curation" group.
To unsubscribe from this group and stop receiving emails from it, send an email to digital-curati...@googlegroups.com <mailto:digital-curation%2Bunsu...@googlegroups.com> .
Kevin S. Clarke
> Thanks Christie, much appreciated. I didn't realize use of JPEG2000 was
> still so low. I would love to hear from people who are using it. Is anyone
> aware of efforts to introduce browser-support?
The places I know using it are using it as the access format (with an
image server) but they store TIFF as the preservation copy.
Kevin
Chris Adams
Edward M. Corrado
format when we acquire digital objects as JPEG2000 files. In other
words, we do not convert JPEG2000 to TIFF or any other format (or
require depositors to do the conversion before providing images to
us). Really, I think a lot of the issues about not using JPEG2000 is
FUD, although I recognize there are some real issues. I agree with
Peter Cliff when he says he feels "that neither [JPEG2000 or TIFF] is
a particular preservation risk at the moment." If JPEG2000 becomes an
actual risk, I believe we will have more than enough time to migrate
the images to something else in the future and see no reason for us to
do it at this time. If that time ever comes, we may have better
options than TIFF.
All that said, we have not gone as far as using JPEG2000 when we
perform our own digitization. We are still using TIFF. We review this
periodically and the next time we do, we may consider changing.
Edward
adam brin
Michael J. Giarlo
substantiating it, does much to further the discussion. Especially when
folks *have* put forward thoughtful arguments against it as a
preservation format. I'd be interested in hearing which of the points
in the LC blog post that was linked earlier ring particularly FUDdy in
your ears.
-Mike
Edward M. Corrado
1. The patent and other legal issues people raise seem to be extremely
overblown.
2. I disagree with the post that JPEG2000 is not widely used. True, it
is not as used as much as other image formats directly on the Web, but
that is a chicken-and-egg issue. However, it is still used (and I
suspect it might be used more than some (non-image) formats that
people think are preservation worthy although I don't have numbers). I
feel it passes the ubiquity test. Of course, this is a personal
decision about how much use makes it common, and you are free to
disagree.
3. Many, if not most major open source image programs support it, as
do many proprietary ones.
3. There are both Open Source and proprietary libraries readily
available. OpenJPEG, for instance is under active development and
version 2.0 was released just this past November and code was added to
trunk as recently as yesterday.
3. JPEG2000 is supported by a number of web browsers including Konquer
and Safari and by Quiktime within other browsers. See the bugzillia
thread linked to in the LC blog post for more details.
4. People can, and have, converted files from TIFF to JPEG2000 and
back without any loss.
Edward
Chris Adams
2. I disagree with the post that JPEG2000 is not widely used. True, it
is not as used as much as other image formats directly on the Web, but
that is a chicken-and-egg issue.
3. Many, if not most major open source image programs support it, as
do many proprietary ones.
3. There are both Open Source and proprietary libraries readily
available. OpenJPEG, for instance is under active development and
version 2.0 was released just this past November and code was added to
trunk as recently as yesterday.
3. JPEG2000 is supported by a number of web browsers including Konquer
and Safari and by Quiktime within other browsers. See the bugzillia
thread linked to in the LC blog post for more details.
Ferran Jorba
after reading for long about JPEG2000 troubles, and suffering huge TIFF
files and serious budget cuts, we are seriously considering migrating
those TIFF to plain JPEGs.
Yes, we know that JPEG means loosy compression. But if it is good
enough for born digital pictures, why cannot it be good for reading old
books, and newspapers, where the original print quality was even lower
than current digitalization quality? It has 100% browser support,
native to all kind of digital gadgets, huge amount of tools, and,
according to our conversions (using ImageMagick default values), 10% the
size of the original TIFF files.
What are we really loosing? Texts pages are crisp, printed pictures are
clear, there are tools to lossles rotate (http://jpegclub.org/jpegtran/)
or all kind enhancements with ImageMagick and other alternatives.
Documents and its information is prefectly kept. And, as David
Rosenthal always stressess, the big preservation issue is economic, not
technical.
10% the size is a strong economic argument, isn't it?
Best regards,
Ferran Jorba
Universitat Autònoma de Barcelona
adam brin
- If we return to OAIS, we have the SIP, the DIP, and the AIP: the original version of the file, a derivative or accessibility version, and the archival version. Does the archival version have to be as accessible as the derivative version? It may be that TIFF, ODT, or PDF/A in 200 years are perfectly reasonable archival formats, but aren't at all accessible. With that said, they were or may have been popular in their time, and are open and viable archival formats. Is it okay if an archival format is less accessible?
- I think the attraction of JPEG2000 is particularly strong because, for example, we have thousands of JPEGs and on average we see between 10 and 100 times the filesize when stored as a TIFF than as a JPEG, and a much smaller ratio when stored as a JPEG2000. (that's likely many TB of archive storage). To put the question a different way… over a 30 year period, is it worth the extra effort to deal with the JPEG2000 file than the overall cost of decoding the JPEG2000, and remember we still have the JPEG. While costs of space have come down significantly, they are still likely save staffing one of the higher costs we have when you factor in backups and other infrastructure dependencies.
Cliff, Peter
Just wanted to add, Oracle long ago dropped the Java Advanced Imaging API and with it JPEG2000 support in Java. In theory this was adopted as a java.net project, but the encoder/decoder has not been touched for some time.
An alternative, jj2000, is also seemingly without a home:
http://jpeg2000.epfl.ch/ (Doesn’t work)
And the fork at:
http://code.google.com/p/jj2000/
Seems just as defunct (check the issues), albeit accessible.
An active imaging library for Java, Apache Commons Imaging (http://commons.apache.org/proper/commons-imaging/) (formerly Sanselan) does not (currently) support JPEG2000 and I don’t see any feature requests to make do so.
I’m not coming to any conclusions here, but pointed out another example where major vendors are failing to see the need for JPEG2000.
Pete Cliff
Digital Preservation Technical Lead
From: digital-...@googlegroups.com [mailto:digital-...@googlegroups.com] On Behalf Of Chris Adams
Sent: 18 March 2013 23:15
To: digital-...@googlegroups.com
Subject: Re: [digital-curation] TIFF vs. JPEG2000
On Mon, Mar 18, 2013 at 5:52 PM, Edward M. Corrado <ecor...@ecorrado.us> wrote:
--
Chris Adams
Randy Fischer
> example, we have thousands of JPEGs and on average we see between 10 and
> 100 times the filesize when stored as a TIFF than as a JPEG, and a much
> smaller ratio when stored as a JPEG2000. (that's likely many TB of archive
> storage). To put the question a different way… over a 30 year period, is it
> worth the extra effort to deal with the JPEG2000 file than the overall cost
> of decoding the JPEG2000, and remember we still have the JPEG. While costs
> of space have come down significantly, they are still likely save staffing
> one of the higher costs we have when you factor in backups and other
> infrastructure dependencies.
using compressed TIFFs, so I suspect that's part of the reason for the
1-2 orders of magnitude difference. If that's the case, I think you
would be better off using TIFF with lossless compression than JP2K.
As far as costs, scaling of staffing and infrastructure effort is the
one big idea that are cloud-based storage systems. It does require a
certain amount of devop knowledge from us end-users, to be sure.
-Randy Fischer
Simon Spero
I'm curious about this: typically preservationists shy away from using compressed TIFFs, so I suspect that's part of the reason for the 1-2 orders of magnitude difference. If that's the case, I think you
would be better off using TIFF with lossless compression than JP2K.
Ferran Jorba
we have more than 670,000 unique TIFF files that occupy more than 15 TB.
We store two copies of them each one in a 25 TB Nexsan Satabeast. The
yearly maintenance of those Satabeasts is now out of our reach, with the
current budget cuts. If we convert (in fact, we have almost finished,
using ImageMagick default values) them into same-quality JPGs, we would
use less than 2 TB space, that we could easily host in corporate
servers, and duplicate somewhere else if needed.
My question stands: what are we _really_ loosing? The data lost with
the loosy JPEG algorithm doesn't affect the _information_ stored in
those files: the text and the images. Why does the library community
insist on those huge files if, for the same storage money, have 10 times
more documents? (I know that I have done a simplification here.)
Any person needing to work on the original high-resolution files
appreciates if them are in JPEG, and not in TIFF.
Who are we working for?
Ferran Jorba
Universitat Autònoma de Barcelona
>
> Hello Ferran - I'm curious how many TB of image files you're
> maintaining and where the costs are piling up. My own institution
> operates on razor-thin margins, so I'm sympathetic to the cost
> concerns, but even cutting 90% of my storage 3TB bill wouldn't
> generate enough savings to do anything else really useful - we
> couldn't hire a programmer or buy much of anything else, for instance,
> and to the extent we're just trying to watch the bottom line, it's not
> a significant enough savings to bother with the risks and conversion
> process. If its'a question of storage and computation across a large
> set of images, though, then I can see where the JPGs might make a case
> for themselves.
>
> --Jacob Nadal
Ferran Jorba
[...]
against common expectations, JPEG compresses much better than JPEG. I
attach and old chart with the comparison that I did for an internal
presentation. JPEG is, for this kind of documents, 10 times smaller
than uncompressed TIFFs, while PNG 75%. Even for grayscale documents
JPEG compresses very well, without any noticeable quality loss. Using
ImageMagick, at that time I wasn't able to create any JPEG2000 version
of my files, after many trials. Finally, I gave up.
How can we trust our collective memories and documents to something so
esoteric and minoritary?
Nathan Tallman
Ferran Jorba
but, again, _what_ are we preserving? The _information_ stored in that
paper, in high quality (say, 300x300, full color, good enough for
anybody interested in the information stored there) or the state that
that particular paper copy had at that time as seen by this optical
capture device?
The information that you say that is lost in a loosy compression has
been already lost if you are interested in the kind of ink or the paper
fiber or the dust stored there. It's gone already! If a particular
researcher needs to study the kind of paper used for this document, he
or she will need the original paper anyway.
Does the cost of this storage, backup, electricity, etc., justify the
extra details that are lost in a typical JPEG loosy compression that, as
everybody agrees, are beyond the average human eye? According to the
output of ImageMagick identify command or JHOVE, the difference between
the original TIFF or the JPEG is negligible.
I think it is disproportionate. Even the best artistic pictures are
stored in JPEG. Why do we need more detail for text (old newspaper)
scans? The images in those old pressess had much, much less detail.
Ferran
Jim Safley
High quality images preserve artifacts that can be lost during lossy
compression. This is especially true with manuscripts. I've come to
appreciate this in my work for the Papers of the War Department. [1]
We house thousands of hand-written scanned documents, and if we didn't
preserve high quality images we'd lose potentially important artifacts
such as watermarks, embossing, fingerprints, light writing, pen
impressions, smudges, stains, and ink bleed-through.
Jim
[1] http://wardepartmentpapers.org/
Ferran Jorba
JPEG loosy compression looses what you won't notice. I do agree to
preserve high quality images, of course. But I'm questioning that there
is 10 times more information in a TIFF file than in a JPEG one. Maybe
there is a tiny fraction more. Does this tiny fraction more information
more cost 10 times more? Again, we are talking about retrospective
digitalization, where the paper is there.
I did some experiments converting full color, grayscale and B/W TIFF
files into JPEGs some time ago and the results are eloquent. Tiny
sizes, no noticeable quality differences. You can check them yourself,
if interested:
http://ddd.uab.cat/record/59776
In our repository, we have been keeping 15 TB of high quality original
digitalisation scans (besides the public PDF version). We have been
asked a few times the original ones, but no one has specified that
wanted the TIFF version, even if we have told them. JPEG has proven
enough for all cases.
I suspect that this TIFF (or JPEG2000) requirement is more a
fear-and-doubt helmet that we, the library community, has agreed to wear
that few people, out of our corporate group, would agree about. If we
ask our user whether they prefer 10 times more documents in JPEG or 10
times less in TIFF, maybe we would have a few surprises. (Again, there
is a simplification here, I know, but the argument stands.)
Ferran
Creighton Barrett
Ben O'Steen
jjn
Simon Spero
Ferran Jorba
I'm answering to all, as your answers have some common points, mainly:
(a) it is not the same preserving the high-resolution image of a printed
document than a unique (archival) one, (b) one should take into account
the purpose of the digitalisation, (c) what do we realy loose using a
loosy algorithm, and last but not least, (d) the economic impact of
choosing one or the other.
I think that the campaign that David Rosenthal has been pursuing in his
blog and talks in the last years against the myth of the digital
obsolescence risk (like, among so many others, in
http://blog.dshr.org/2007/04/format-obsolescence-scenarios.html, up to
the last one so far,
http://blog.dshr.org/2013/02/rothenberg-still-wrong.html) could partly
be taken to this loosy compression disaster scenario (sentence from this
last post):
"how can we use our limited resources to maximize the value delivered
to future readers?"
I'd like to know a real case of a
slightest-colour-variations-may-lead-to-further-discoveries (Ben's
answer) that goes beyond the theoretical case, and where the *format*
choosen for this file makes *the* difference. On printed material, the
difference between my copy or your copy of a 19th century newspaper is
irrellevant (given that they are in good enough conditions, of course).
The important is the what was published there, not if my copy is
slightly more yellow than yours. Again, I'm thinking as a librarian,
here not as an archivist.
I'm using paper copies as model: if a library gets a better copy of a
printed book (without unique marks, like valuable autographs, that is),
it replaces it, because the important think is that the readers have a
good copy. The individual characteristics or damages of this particular
paper book are not relevant for the users.
Some of the examples against JPEG, like Ben's jpeg resaving link are,
I'd say, misleading
(http://www.impulseadventure.com/photo/jpeg-resaving.html). The
original high-resolution copy has to be kept intact. If somebody
alters-recompress-watermarks-saves-recompress is doing a bad
manipulation of the file, as it is a bad manipulation of the paper to do
it with dirty hands. This doesn't go against paper (I can imagine the
headers: "Paper can get dirty, paper is a poor information medium!"),
but against the person that manipulates valuable material without
washing his or her hands. If we keep the original high-resolution JPEG
intact, we are safe against poor manipulations.
Instead, I'm facing webs, like Jim's War Departments Papers
(http://wardepartmentpapers.org/) where the papers have been scanned in
B/W or grayscale:
http://wardepartmentpapers.org/docimage.php?id=3958&docColID=4262
Maybe the high resolution of those scans are kept in TIFF, but *most* of
the data is already lost when it was decided to scan in grayscale
instead of full color. According to my experience, a full color JPEG is
smaller than a uncompressed grayscale TIFF. Do historians really prefer
those grayscale *because* they are in TIFF, instead of full color *even
though* they are in JPEG?
I think that we, as community, have to think twice, after all those
years, about our recommended practices, if we want to be useful to the
society and the future.
Thanks for your answers,
Andrew Woods
One discussion point that seems to have been overlooked thus far is
the actual robustness of the various image formats in the face of bit
rot. The following article takes a systematic approach at analyzing
the resilience of JPEG, TIFF, and JPEG2000 to increasing percentages
of corrupted bits.
http://www.dlib.org/dlib/july08/buonora/07buonora.html
That is all to say that file size and data loss during digitization
are only two dimensions of consideration. Depending on the
preservation environment anticipated for the digital artifacts, format
robustness should be taken into consideration as well.
Andrew
Ferran Jorba
bit rot consideration is critical if there is only a single copy of the
file. If there is more than one, and a checksum value somewhere to
choose the right one, and/or a parity file to correct the missing value,
it becames less critical. As a matter of fact, my library cannot
expect an external mirror of our high-quality TIFF files if they occupy
16 TB, but we can easily find an external site for 2-3 TB if they are
in JPEG. So, the bit-rot argument can be defeat easily, it becomes a
self-correcting problem.
But, anyway, at our university we have been keeping digital
documents (PDFs version 1.0) since 1996, that have been migrating from
server to server, and I've never found an unreadable file. Since the
last 6 years, we also keep the md5 and sha1 file of our 1,3 milion
files (counting TIFFs, the corresponding JPEGs and thousands of PDFs)
and, again, no checksum has ever failed. And, I can assure you, we
have not been using high-end equipment at all.
My two cents experience,
Ferran Jorba
Universitat Autònoma de Barcelona
Andrew Woods <awo...@duraspace.org> escrigué:
Matt Schultz
I've really enjoyed this thread. You've raised brave questions and have given everybody a good example of how an institution analyses a difficult problem in the context of their local resource constraints. Some of the best comments from folks I think have been the ones who have boiled these sorts of decisions down to use cases and local needs. Which is not to say that that is where the conversation should stop necessarily - the tension between local preferences and technical concerns probably warrants some more great expert comments.
The backdrop for all of your decision-making is important to me. As I've read it, you are an institution that suddenly finds yourself with an abundance of data and in need of making some quick decisions about how to continue supporting it. It is yet another reminder to me of how important it would be for all of us in the cultural community to start building toward some best practices literature aimed at the early stages of creation, selection and acquisition (of both born-digital and digitized content)--getting us all thinking more seriously about the consequences of the decisions we make at those early stages for the long-term stewardship of the data.
Who is my current Designated Community? Who are they likely to be in 5-10 years? Why am I creating all this metadata right now? What am I really going to do with it? How will this collection grow? What is my collection policy and how will this type of data likely accumulate in the next 5-10 years? Etc., etc. Digitization and Digital Preservation I think are too divorced from one another in this sense and yet often times conflated to the detriment of both ends of the lifecycle.
If anybody knows of such literature out there I would welcome the pointers.
In any case - I have found this discussion to be really useful so far. Thanks!
Matt Schultz
Program Manager
Educopia Institute, MetaArchive Cooperative
http://www.metaarchive.org
matt.s...@metaarchive.org
616-566-3204
jjn
Ferran Jorba
many years ago, a professional musicial I had the luck to know, told me
something I've never forgotten, because it impressed me so much: "When I
listen to music, often I'd wish I knew less about the technicalities of
music, because then I cannot enjoy it".
Many times I read about digital preservation papers and discussions, I
feel that going so much into the technical details misses our goal as
digital preservationists. Or digital librarians; we don't call
librarians "paper preservationists", do we?
Thanks,
Ferran
El Fri, 22 Mar 2013 12:41:23 -0700 (PDT)
jjn <jna...@gmail.com> escrigué:
Ferran Jorba
thanks for your answer. I'd like to comment a little bit about the last
point you've made:
Digitization and Digital Preservation I think are too divorced from one
another in this sense and yet often times conflated to the detriment of
both ends of the lifecycle.
like TIFF and JPEG2000 happens almost exclussively in the
ditigalization circles. And it is precisesly in digitalization where
the paper is (still) there, and in a significative number of times, in
many copies, as long as it is printed material. And, to bring even
more contradiction to the argument, so many times the capture was done
en grayscale or, even worse, in black and white, where the loosiness
whas perpetrated in the fist stage.
Myself, being an affectionate reader of old material, feel that black
and white is difficult to read, grayscale is depressing to read, and
color capture a joy to read. And the argument for not taking color
capture often is that color capture produce large files that are
expensive to keep, but insist on TIFF! Please, let's go down to
earth! I'm part of those circles, but being also part of the readers,
I'd like the community to balance more to the readers and less to the
specialists that are confusing us with too much technical details.
Paper librarians are not chemists, and they they have been doing
perserving paper documents job for centuries. They need to know about
humidity and temperature conditions, some weight calculation and lots
of common sense.
About the "preservation role" that often is associated to those formats,
again, I'd like to quote David Rosenthal:
This suggests that the idea of "preservation formats" as opposed to
"access formats" is a trap. Precisely because they aren't access
formats, preservation formats are less likely to have the strong open
source support that enables successful preservation.
http://blogs.loc.gov/digitalpreservation/2013/01/is-jpeg-2000-a-preservation-risk/#comment-11802
Now, I'm not thinking in my institution. Internet has no borders. I'm
sad that so many documents have been digitalized in poor legibility
conditions due to format fundamentalist reasons.
Thank you all for your contributions,
Ferran
El Fri, 22 Mar 2013 15:21:16 -0400
Matt Schultz <matt.s...@metaarchive.org> escrigué:
Declan Fleming
Ed Fay
Ed Fay
Collection Digitisation Manager
Library, The London School of Economics and Political Science
10 Portugal Street, London WC2A 2HD
E....@lse.ac.uk
0207 955 7235
http://www.library.lse.ac.uk/
http://www.twitter.com/digitalfay
Tito Robe
Hello everyone, interesting thread, but no one has mentioned if is possible convert from each other, for example right now I need to convert from TIFF to JPEG2000 in Java, but I haven't been able to find any documentation about that, even in this site.. where you have talked so long about that subject. If someone has done something like that.. I would thank a lot any help..
Tito Robe
Kevin Hawkins
About convert from TIFF to JPEG2000 in Java
Hello everyone, interesting thread, but no one has mentioned if is possible convert from each other, for example right now I need to convert from TIFF to JPEG2000 in Java, but I haven't been able to find any documentation about that, even in this site.. where you have talked so long about that subject. If someone has done something like that.. I would thank a lot any help..
--
You received this message because you are subscribed to the Google Groups "Digital Curation" group.
To unsubscribe from this group and stop receiving emails from it, send an email to digital-curati...@googlegroups.com.
To post to this group, send email to digital-...@googlegroups.com.
Visit this group at http://groups.google.com/group/digital-curation.
For more options, visit https://groups.google.com/d/optout.
Tom Creighton
We chose JPEG2000 vs. staying with TIFF because it is widely supported and understood and we had access to acceptable software libraries. We believe the standard has longevity. Also, without losing quality, representing the images as JPEG2000 with lossless compression changes TIFF images that are on the order 10-12 MB each to 5-6 MB each for JPEG2000. We process about a million images per day and already have a repository storing in excess of 10 PB of images for one copy. The cost savings are significant.
// Create the Image Plan
List<Operation> plan = new ArrayList<Operation>();
Operation op = new Operation(Operation.Type.DECODE);
op.addParameter(“inputimage”, “img0”);
op.addParameter(“outputimage”, “img0”);
plan.add(op);
op = new Operation(Operation.Type.ENCODE);
op.addParameter(“inputimage”, “img0”);
op.addParameter(“outputimage”, “img0”);
op.addParameter(“format”, “jp2”);
op.addParameter(“quality”, 100);
plan.add(op);
// Convert the image
byte[] img = // Read the input image into this byte array
ImageLibrary lib = ImageLibraryFactory.createImageLibrary();
int planId = lib.createOperationPlan(plan);
Map<String, ConvertResult> cnvResult = lib.convert(planId, img);
ConvertResult jp2 = cnvResult.get(“img0”);
// Lossless JP2 is found in byte[] in jp2.img
FileOutputStream fos = new FileOutputStream(xxxx);
fos.write(jp2.img);
fos.close();
// Cleanup
lib.deleteOperationPlan(planId);
Hope this is useful.
tc
Tom Creighton
CTO FamilySearch
Tito Robe
You received this message because you are subscribed to a topic in the Google Groups "Digital Curation" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/digital-curation/jX1MELYvpKE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to digital-curati...@googlegroups.com.
Kevin Hawkins
http://en.wikipedia.org/wiki/JPEG2000#Applications . --Kevin
> changes back.
>
> Encoding/decoding JPEG2000 we use the commercial library kakadu
>
> I recommend finding a way to call ImageMagick from your Java
> application. —Kevin
>
>
> On 4/30/14 9:29 PM, Tito Robe wrote:
>> Hello everyone, interesting thread, but no one has
>> mentioned if is possible convert from each other, for
>> example right now I need to convert from TIFF to JPEG2000
>> in Java, but I haven't been able to find any documentation
>> about that, even in this site.. where you have talked so
>> long about that subject. If someone has done something
>> like that.. I would thank a lot any help..
>>
>>
>>
>>
>> --
>> You received this message because you are subscribed to the
>> Google Groups "Digital Curation" group.
>> To unsubscribe from this group and stop receiving emails from
>> it, send an email to
>> digital-curati...@googlegroups.com
>> <mailto:digital-...@googlegroups.com>.
>> http://groups.google.com/group/digital-curation.
>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the
> Google Groups "Digital Curation" group.
> To unsubscribe from this group and stop receiving emails from
> it, send an email to
> digital-curati...@googlegroups.com
> <mailto:digital-...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to a topic in
> the Google Groups "Digital Curation" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/digital-curation/jX1MELYvpKE/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> digital-curati...@googlegroups.com
> <mailto:digital-...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Digital Curation" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to digital-curati...@googlegroups.com
Tom Creighton
Chris Adams
Jon Stroop
https://code.google.com/p/openjpeg/
Also, I believe that since version 6.8.8 ImageMagick is using OpenJPEG instead of Jasper, at least that's what I assume from the evidence in these threads:
http://imagemagick.net/discourse-server/viewtopic.php?t=25357&p=109914
http://imagemagick.org/discourse-server/viewtopic.php?f=3&t=25362
-Jon
Chris Adams
Carl Fleischhauer
Warning: this is a _lengthy_ set of, um, musings, devoid of the helpful and specific technical notes (and even examples of command-line software instructions!) that characterize many of the TIFF-vs.-JPEG2000 comments that preceded this one. Skim-reading is advised.
I send my thanks to the many commentators--including my Library of Congress colleague Chris Adams--for this illuminating thread! I sometimes deliver a stump speech about the lack of consensus about digital preservation practices for video and motion picture film scanning. In the speech, I compare this lack of practice-maturity to the relatively more mature and settled state of digitization that employs still imaging or waveform sound formats.
In the face of this thread, however, my stump speech seems to be a bit of an overstatement. I should know better: I help out with the FADGI Still Image Working Group, and see first hand the still-evolving matter of imaging performance metrics (and related tools) and the slow implementation of fully realized color management in memory institutions. These are two more indicators of the not-yet-settled-ness of still image digitization. (I won't even mention the background topic of embedding metadata and the use of image-level identifiers. Yikes!)
Regarding JPEG 2000 in the still image realm, I echo Chris's frustration over the state of tools: it is maddening that more have not come into play, and Chris is also correct that we ought to collectively put more resources into the game, especially supporting the development of open source tools.
There is a bit of a chicken-and-egg to this, as one commentator noted: with more adoption, more tools would come along; with more tools, we'd have more adoption. Sigh. It is tempting to speculate about the underlying causes, which have nothing to do with image quality: everyone agrees that wavelet transforms (JPEG 2000) provide results that are visually superior to DCT applied to 8-pixel blocks (old JPEG). And I agree with the commentator who wrote that patents are also not a real issue.
Chris mentioned one yardstick for adoption: "Will it play in a browser?" There is a second yardstick that also counts for a lot in our age of ubiquitous digital photography and the social Web: "What formats are natively supported in the camera?" Camera specifications are the outcome of decision-making in Japan, supported by JEITA. The old JPEG format has proven to be good enough for most photographers (in contrast, the pros shoot raw) and JPEG's wide support in many applications leaves camera manufacturers in no mood to change.
A few years ago, I attended a professional photography conference at Microsoft that included strong advocacy for their then-new Windows photo format (subsequently called HD Photo and now standardized as JPEG XR). Like JPEG 2000, this Windows format was clearly technically superior to old JPEG. But the sense of the conference was that if it didn’t get into cameras as a native format, there was little hope of adoption. It is hard not to be reminded of the frustration we hear from our colleagues in professional sound recording (including Neil Young and his push for his Pono format): "How can people stand to listen to those awful MP3 files?" Yet MP3 is everywhere and it passes the "good enough" test for the masses.
It has been interesting to compare the relatively warm embrace that JPEG 2000 has received in moving image quarters with the cold shoulder we see for still images. The Digital Cinema Package (DCP) is _the_ widely adopted format (truly universal these days) for the theatrical distribution of motion pictures and it employs lossy JPEG 2000 compression. To be sure, this is a closed system in which the content creators have dictated the specification to theater operators, and projector manufacturers are required to have their systems tested for conformance. But the very high level of clarity in the lossy JPEG 2000 imagery--thanks again to wavelet transforms--was very important to the motion picture industry. The picture has to look good on the screen; in comparison, Internet-delivered MPEG is still pretty awful. For theaters, JPEG 2000 is unshakably in place.
We also see the adoption and use of JPEG 2000 elsewhere in professional moving image circles, ranging from digital moving image cameras to post-processing systems. Beginning in 2007, the Library of Congress began reformatting its holdings of older video recordings into a format that employs lossless JPEG 2000 compression, and we now have tens of thousands of digital video master recordings in this format, petabytes-worth. My sense is that there is a bit of open source development to support JPEG 200 with moving images but, as noted, the adoption has mostly been among professional users, and these are folks who are very accustomed to purchasing commercial products.
There are a couple of other considerations to throw into this idea marketplace. One came up at our last FADGI still image meeting and was echoed in the flow of comments in this Digital Curation thread. At the FADGI meeting, we heard from a representative from a federal agency that partners with commercial entities. One of their partners scans microfilmed census and other historical records using an all-JPEG-2000 workflow, producing millions of images every year, which are then given to the agency. I don't remember the details, but my impression is that the commercial partner produces JPEG 2000 master images with a modest level of lossy compression. That would be easy to believe: wavelet transforms in modest-lossy compression will retain the legibility even of handwriting. Some folks even argue that a lot of what is discarded in modest-lossy JPEG 2000 compression is noise, thereby (according to this account) improving the image quality. In any case, for these forms of content, neither the originals nor the microfilm masters are being discarded, so the anxieties about authenticity come down a notch.
Meanwhile, I have been told that one or more of the so-called book mass-digitization projects also use lossy-JPEG-2000 workflows. The critiques of those projects tend not to complain about technical image quality but rather about "housekeeping": missing pages, pages out of sequence, thumbs or fingers obscuring text, or blurring caused by a page flipping or moving when being scanned. When the thumbs are out of the shot, the JPEG 2000 imagery delivers a perfectly readable typographic page. And some presentation applications take advantage of JPEG 2000's tiling options to support zooming in the browser, as the Library does for, say, its map collections. (This happens under the covers since, as has been noted, JPEG 2000 imagery cannot be directly displayed in browsers.)
At our FADGI meeting, our colleague's remarks did not concern JPEG 2000 as an output for the agency's own image production; rather, the question had to do with the receipt and preservation of JPEG 2000 images delivered to the agency under the terms of the partner agreement, images that the agency can make publicly available in a few years time. This circumstance--JPEG 2000 images delivered by a partner--also arises in some libraries as a result of their partnering in mass book-digitization partnerships. The point is that some organizations have, ipso facto, large bodies of JPEG 2000 imagery that they need to be able to manage over the medium or long term. This is a just another reminder that many archives must support the long-term management of more than one digital format.
The topic of when and where to use JPEG 2000 at the Library of Congress for still image digitization comes up from time to time in informal conversations. (As noted, this is pretty much a settled matter for moving image materials.) Alas, our informal conversations have thus far produced little more consensus than what is manifest in this thread.
These informal conversations sometimes conflate a pair of topics, the second one being whether, where, and when to accept lossy compression for master files. Some have looked at classes or categories of material, for example (a) catalog cards, (b) widely held twentieth-century printed matter (think "congressional documents"), (c) maps, (d) manuscripts, and (e) photographs (including negatives). Reading from left to right, one can make the argument that the imaging stakes go up in a progressive manner. Might lossy compression be acceptable for the first one or two in the spectrum, and not for the others? The Library carried out a bit of an exploration of the lossy-vs-lossless question in the mid-1990s but the findings did not take root (http://memory.loc.gov/ammem/pictel/). That twenty-five-year-old project compared old JPEG to uncompressed images and, for certain classes of manuscript, lossy seemed acceptable. But the context was different. For example, we were then shy about color--we remembered that monochrome had always been accepted for preservation microfilms and we worried about big files. Today, I think we would likely embrace color for most manuscripts and, if we still accepted lossy, we would be glad to see the added clarity provided by JPEG 2000.
At the risk of forking this thread into a tangent, I would like to ask for experience narratives about the use of PNG, a format that Chris mentioned. For several years, due to inattentiveness on my part, I had relegated PNG to the use case of access-via-browser. This was due to the fact that the format was initially created as a reaction to the threats of licensing fees for GIF (another good-for-browsers format) in the 1990s. Last month, however, I re-read the W3C specification for PNG and found lots of nifty features, on paper at least. For example, there are specs for things that provide helpful support for color management, including a group of tagged color-space metadata elements document things like primary chromaticities and white point, image gamma, and carry an embedded ICC profile. In addition, as has been reported in this thread, PNG offers lossless compression with good results (albeit apparently a bit less efficient than JPEG 2000). Do any libraries or archives that use PNG as a mastering file? Have people found that tools support some of the features that caught my eye, like the ones that support color management?
Whew! Thanks for your patience with this long posting.
Carl Fleischhauer
Simon Spero
--
Jay Gattuso
"Anyone got some hi-res scans of newspapers?"
Hi Simon,
Do you still want some high-res scans of newspapers?
From our digital newspaper archive http://paperspast.natlib.govt.nz/ you can get to more high-res lossless png scans than you'll ever need.
We have some helper methods to help get to the content if this is still an active project / concern for you.