Revitalizing an Old IMG/FIT Multithreaded Converter

[Kevin McAbee]

Hello!

I have a question about location and differences of the JAVA api from the new VICAR repo on github versus the old repo (can't remember where I found it years ago), that was Java 1.4 based (iirc). But first, context.

Long ago, in a land far far away (2012), I was helping a filmmaker out on an upcoming IMAX film based on space mission imagery. I wrote a multithreaded Java app to scrape released archives of image data files of the Cassini mission on PDS. Then another app opened those IMG files, and converted them to 32-bit TIFF (with IMG labels and data as exif) after running IDL calibration code (that I ported to Java) against it. I also added further plugins to read FITS files (layered on nom.tam.fits) and finally added support for the SDO project.

Coming full circle, I'm looking to revitalize that project, clean it up, port it to Java 8, vastly improve its concurrency, and then open source it. The idea being that the general public, or at least amateur astronomers, can have an easy system to download image files, convert them en masse, and have database access to the image header data. First step? I looked around to see what libraries had been updated in the intervening years. 

Lo and behold, VICAR is now on github with a brand new release! YEEEESSSS! I also see that it has incorporated all those wonderful JAI and other external libraries that were a pain. YEEESSSS! And finally, it's pulled in nom.tam.fits. YEEESSSS!

However, I want to make sure I utilize this behemoth library in a proper manner. I glanced through the quick start guide, and noticed a brief mention of Java dev capabilities, but didn't see anything in depth. Any help, pointers, or conversations would be greatly appreciated.

Also, thank you for making the original repo, and providing this update. Makes me so very very happy to see this all coming together!

Regards,

Kevin

[Bob Deen]

Hey Kevin...

Ask questions and we'll try to answer!  We use the Java imageio mechanism to do format conversion... which we do for every Mars image we process.  We use the transcoding capability of imageio to convert images and preserve metadata.   We convert back and forth between vicar and pds all the time, but others are possible.   You'll probably need to write an XSLT template to add new types of metadata translation... but image translation is simple and Just Works between any of the supported formats (including fits, and all the usual tiff/png/jpg/etc formats).  There are some XSLT examples in the distro, although they are not trivial!

Documentation is not really our strong suit :-(   (a matter of funding and priorities). I think the javadocs are in reasonably good shape but we don't have a lot (any?) of higher level conceptual stuff.  Best bet is to ask questions when you get stuck.

-Bob

Sent from my iPad

--
You received this message because you are subscribed to the Google Groups "VICAR Open Source" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vicar-open-sou...@googlegroups.com.
To post to this group, send email to vicar-op...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vicar-open-source/b09c3626-86fb-451e-af90-00b82b9e066c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kevin McAbee]

Bob,

Sounds good. I have another project I'm working on bringing up to date first - (the downloader/scraper of PDS Cassini folders for IMG and LBL files. I'll then shift over to the Converter itself, which is where I'll hit all the wonderful issues with upgrading all the libraries and APIs.

By chance, are you in contact with the team who heads up the PDS image data system? If so, could you give me a contact there? It might be that there have been enough updates and/or interface changes to not warrant spending the time on updating the Downloader, which recursively runs through folder structure links and downloads LBL and IMG files.

I do want you and your team to know that people do want the data, and they do want to utilize all the things you are providing. It really helps the public at large, especially everybody interested in astronomy. So, thank you for all of your hard work and efforts.

Regards,

Kevin

[Bob Deen]

Hey Kevin...

I recommend contacting Jordan Padams of the pds imaging node (cc'd here) to work that kind of technical detail.

-Bob

Sent from my iPad

[Kevin McAbee]

Bob et al,

Question for you and your team? Which JDK are you using and/or targeting? I've completed the POC of the Downloader rewrite and will be moving on to the Converter (VICAR formats > TIFF/PNG w/ EXIF data from LBL/IMG headers).

In other news, I've pulled in the VICAR project, and have migrated the project strutcture (java only) to the standard structure, and have added gradle and JUnit. I've migrated the Test classes to a standard test project structure, and am currently migrating each of the standalone Test classes to run off JUnit testing. After that, as I work through the conversion code, I'll likely start porting the VICAR code to JDK 8 using its Streams, functional programming practices, etc. Also, as I try to figure out the documentation, I'll likely add that as well.

Would there be any interest from you or your team in creating a separate branch for JDK 8 where I could submit PRs or should I fork off your repo?

As always, thank you for opening up this code to the public.

Regards,

Kevin

On Sun, Sep 11, 2016 at 10:03 AM Bob Deen <Bob....@jpl.nasa.gov> wrote:

[Bob Deen]

Hi Kevin...

Currently we use Java 7. We haven't had a compelling need to move to 8
yet, and there's something to be said for not getting too far ahead of
the curve. ;-) But primarily, it's an issue of resources... until Java
7 becomes hard to find (as in, vendors no longer support it), or until
we have a compelling use case for some new feature in Java 8, it's hard
to justify the effort given everything else we have to do.

But... that's exactly where you (and others) come in... it's kind of the
point of open source! We would be quite interested in hearing about
what you're doing, how it's working, and whether it improves the code in
some way (performance, maintainability, features, whatever).

We have not yet set up anything formal for back-contributions... again,
it's a matter of resources; we kind of need some back-contributions to
force the issue. ;-) Maybe this is it? Let's talk about how to do
this... we're new at this whole open source thing and don't really know
what we're doing yet.

One of the big concerns we have is that we are supporting operational
missions with this code, so we have to be careful what and how we accept
back into the mainline so as not to disrupt operations. In principle
we're quite excited to get contributions, in practice we have to figure
out how to deal with them responsibly. Again, suggestions/discussions
are welcome!

Thanks...

-Bob

>> <mailto:vicar-open-sou...@googlegroups.com>.

>>
>> To post to this group, send email to
>> vicar-op...@googlegroups.com

>> <mailto:vicar-op...@googlegroups.com>.

>>
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/vicar-open-source/b09c3626-86fb-451e-af90-00b82b9e066c%40googlegroups.com

>> <https://groups.google.com/d/msgid/vicar-open-source/b09c3626-86fb-451e-af90-00b82b9e066c%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Kevin McAbee]

Bob,

Thanks for the quick responses from you and your team. I would welcome the chance to step you and others through both what I'm doing, and my reasoning behind all of the previously mentioned items. I would also love to talk about processes, including Pull Request submissions, branches, issue tagging, and more.

My day job is a Senior Architect for an agency that works with a variety of businesses and vertical markets to provide long-term enterprise scale software solutions or help them iteratively upgrade/improve existing/legacy infrastructures. I've worked at multiple startups, agencies, international corporations, and higher education. 

Why mention the above? To say that after working at all these places and projects, I fully understand and appreciate the need to have proven, solid, working software for mission critical needs. That's why I've been focused on adding a build and dependency management system (via Gradle, though it could just as easily be maven), a full test suite (Junit 4.12), and replacing System.out calls with a logging framework (SLF4j masking Logback) for results and further checks. And, while I've been focused on the Java code, the build system could be utilized for other areas and more.

The JDK8 additions revolve around simplifying and improving data workflow and changes, adding functional programming concepts, and replacing pre-JDK5 sections with newer, recommended APIs, Objects, or methods.

If you'd like, we could get together over some form of web-based communication. My company uses gotomeeting, but there's also Google Hangouts, Screenhero if you use that, joinme, or any number of others. I could show you both my project - the software helping back mission image acquisition and processing for the IMAX film In Saturn's Rings (http://insaturnsrings.com) - and some of the changes I've either made or plan on making.

Thanks and I look forward to hearing from you!

Regards,

Kevin

[Bob Deen]

Hi Kevin...

Just wanted to touch base on this, see if you have any changes you're
interested in submitting back. We don't really know what we're doing
yet in terms of accepting stuff, but this might be a place to start...?

Thanks...

-Bob

On 10/10/16 1:39 PM, Kevin McAbee wrote:

>> <mailto:vicar-open-sou...@googlegroups.com>.

>>
>> To post to this group, send email to
>> vicar-op...@googlegroups.com

>> <mailto:vicar-op...@googlegroups.com>.

>>
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/vicar-open-source/b09c3626-86fb-451e-af90-00b82b9e066c%40googlegroups.com

>> <https://groups.google.com/d/msgid/vicar-open-source/b09c3626-86fb-451e-af90-00b82b9e066c%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Kevin McAbee]

My apologies for the very delayed response. I had to set aside my work, I am interested in helping and contributing back. However, I will need to start over since I didn't (woefully) start committing my VICAR changes until I was well into the process.

I would be interested in talking sometime soon about what I've done and finding out which things you may or may not be interested in accepting.

You received this message because you are subscribed to a topic in the Google Groups "VICAR Open Source" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vicar-open-source/fOCyjYVIpMA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vicar-open-sou...@googlegroups.com.
To post to this group, send email to vicar-op...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vicar-open-source/27a0f88d-b8a6-ba67-84b4-4f621a97b099%40jpl.nasa.gov.