NITF reader status (and inevitable questions)

[Brad Hards]

I've done some more work on the NITF reader. I'm now able to read some more of
(but not all) of the NITF 2.1/NSIF 1.0 files.

Most of yesterday was spent building a test tool that runs across a list of
files. For each file it:
- invokes gdalinfo and saves a (slightly filtered) output to a file (i.e. gdal
as oracle)
- tries to generate the same format using the file reader
- compares the results of our format to the oracle.

Code at https://github.com/bradh/nitf-metadata-comparison

That showed me how far I still have to go...

I'm now starting to read extensions. The current design approach is to build a
parser for, and then use, the GDAL TRE data
(http://trac.osgeo.org/gdal/browser/trunk/gdal/data/nitf_spec.xml)
Still working that through.

I have some concerns about how to present that data though. That is, what API
would you like?

I've though of some alternatives .

One idea is a flat key/value pair (Map<String,String>), where the key contains
a concatenated TRE name and field. This would be conceptually similar to the
approach used by GDAL (although I'd probably lose the NITF_ prefix).
NITF_PIAIMC_CAMSPECS=
NITF_PIAIMC_CLOUDCVR=010
NITF_PIAIMC_COMGEN=00
NITF_PIAIMC_ESD=Y
NITF_PIAIMC_GENERATION=1
NITF_PIAIMC_IDATUM=ZYX
NITF_PIAIMC_IELLIP=RF
NITF_PIAIMC_MEANGSD=00098.4
NITF_PIAIMC_PREPROC=P1
NITF_PIAIMC_SATTRACK=00000000
NITF_PIAIMC_SENSMODE=PUSHBROOM
NITF_PIAIMC_SENSNAME=PRISM N
NITF_PIAIMC_SOURCE=PROCESS:JAPAN-JAXA-EOC-ALOS-DPS 20090107065215
NITF_PIAIMC_SRP=Y
NITF_STDIDC_ACQUISITION_DATE=20080223
NITF_STDIDC_COUNTRY=
NITF_STDIDC_END_COLUMN=005
NITF_STDIDC_END_ROW=00016
NITF_STDIDC_END_SEGMENT=AA
NITF_STDIDC_LOCATION=3416N13227E
NITF_STDIDC_MISSION=ALOS
NITF_STDIDC_OP_NUM=000
NITF_STDIDC_PASS=
NITF_STDIDC_REPLAY_REGEN=000
NITF_STDIDC_REPRO_NUM=00
NITF_STDIDC_START_COLUMN=001
NITF_STDIDC_START_ROW=00001
NITF_STDIDC_START_SEGMENT=AA

Alternatively we could build a tree of TREs (no pun intended), where the top
level key would be the TRE name (e.g. PIAIMC), and the value would be another
map (from field to actual value, e.g. SENSMOD:PUSHBROOM).

Both of these flatten nested/looped structures (e.g. CSEPHA is going to have
stack of fields that look like EPHEM_0_X, EPHEM_1_X, EPHEM_2_X and so on.

Another approach would be to open code every known TRE (e.g. using an offline
code generator tool that makes accessors and the parsing code). That could
give us a better representation of the TRE structure, but would probably be
harder to maintain, and would prevent adding new TRE structures at runtime.

The final concept was to say to "its all out of scope, here is the raw data as
a string, parse it yourself". Seems weak though.

Brad

[Michael Menousek]

Again, awesome progress Brad. I think that flat key/value is going to work
best. I think that the NITF_ prefix is redundant as well, but eventually
we'll need a way to scope our keys to a particular taxonomy. This is
quickly getting us into that area. Would "NITF:PIAIMC_CAMSPECS" be okay?
We'd also need a way to validate the format of such a field. I fully
realize this is getting into other areas and the quick answer to you is to
use something like NITF:XXX_YYY. We may also need to show how to put
hierarchical key/value into a structure that is flat. Looks like you
already hit that with things like EPHEM_1_X, EPHEM_2_X.

One clarification question: we aren't actually going to use GDAL right?
It's still going to be pure Java?

I should note that this is my initial reaction and I'm still very
interested in what the rest of the group says.

Thanks again. Really excited about this.

Michael

>--
>You received this message because you are subscribed to the Google Groups
>"ddf-developers" group.
>To unsubscribe from this group and stop receiving emails from it, send an
>email to ddf-developer...@googlegroups.com.
>For more options, visit https://groups.google.com/d/optout.

[Brad Hards]

On Wed, 2 Jul 2014 01:06:23 AM Michael Menousek wrote:
> One clarification question: we aren't actually going to use GDAL right?
> It's still going to be pure Java?

Yep. The only bit of GDAL I'm planning to borrow / copy is that XML file and
possibly the associated XSD, which will probably live as a resource in the
jar.

Brad

[Jeff Vettraino]

Hey Brad - Can you clarify what you are talking about in regards to API?  Is this a specific API that users would use when only dealing with NITFs (or similar data)?  Or is this an nitf/image specific extension to the Metacard interface?

If it is the first, I (personally) always lean towards an explicit API meaning having methods for the specific fields that you would be exposing through that API, rather than a Map that you then have to know the keys to pull out the Map anyways.  I think the explicit APIs are more user friendly and are easier to use.  All that being said I don't know a lot about the TRE data so if it is something that needs to be truly dynamic at runtime, then the map might have value.

If you are referring to the Metacard, then I would think that most of the data you listed you would want to put in the METADATA attribute as XML, and then make explicit attributes for items that you would envision clients needing to search upon.

Hopefully I am not completely missing the mark in regards to the API you are referring to.

[Brad Hards]

On Tue, 1 Jul 2014 06:24:07 PM Jeff Vettraino wrote:
> Hey Brad - Can you clarify what you are talking about in regards to API?
> Is this a specific API that users would use when only dealing with NITFs
> (or similar data)? Or is this an nitf/image specific extension to the
> Metacard interface?

The former. We still need to work out the exact metadata representation in the
Metacard, later.

> If it is the first, I (personally) always lean towards an explicit API
> meaning having methods for the specific fields that you would be exposing
> through that API, rather than a Map that you then have to know the keys to
> pull out the Map anyways. I think the explicit APIs are more user friendly
> and are easier to use. All that being said I don't know a lot about the
> TRE data so if it is something that needs to be truly dynamic at runtime,
> then the map might have value.

I agree with the general sentiment, but the concept behind is TRE is
extensibility (see MIL-STD-2500C Section 5.8 for the detail, especially around
Registered vs Controlled Extensions). There is a register (see
http://jitc.fhu.disa.mil/cgi/nitf/registers/alpha.aspx#A) and we could
explicitly encode all of those (manually or using some code generation), but
for some of them (e.g ACIMAT), there is intended meaning (some kind of
acoustics thing) but only to a particular community of interest. For some
(e.g. OMCPHA), its either obscure or (more likely) sensitive enough that JITC
won't publish the field. So explicit API would be possible for some, but
certainly not all, of the TREs.

There is another issue - no single NITF file is going to use every TRE. So for
a metadata collection, we'd need to either walk the entire API, or keep some
kind of list of what TREs are available at each part of the file (e.g. top
level, each image, each graphics segment, each text segment).

> If you are referring to the Metacard, then I would think that most of the
> data you listed you would want to put in the METADATA attribute as XML, and
> then make explicit attributes for items that you would envision clients
> needing to search upon.

Do you have an XML schema and a list of attributes you'd like to see?

Brad

[Jeff Vettraino]

Thanks for the details and the link Brad.  Certainly helps a bit.  Would inheritance be a fit here?  With this being a complicated concept/area the more user friendly the API, the better (so it will be easier for people like me who don't have a deep understanding of TRE to use).  What do other NITF parsers do to handle these cases (or do they not handle them)?  My other thought is (which may or may not be contradicting what I previously said), keep it simple, especially at first.  So I don't know what the use case you have in mind that would leverage this API (seems like a client app that specifically uses NITFs).  If you don't have a specific app or use case in mind then I would focus on parsing out the metadata for exposure in the Catalog.

Along that train of thought, I think being able to provide a thumbnail(s) would be extremely beneficial (more so than the specific NITF details).  Also I will post a list of attributes that I think would be great to include in the XML, however I don't have a schema (but will ask around).

[Michael Menousek]

While I like a well-defined API, I like a flexible and future-proof API more.

I think that one "win-win" option here is to provide an embeddable JAR that client classes could use to easily extract NITF info from a NITF Metacard.  See, the problem with the API discussion is that the component (class) only gets a Metacard, and we need the data to be usable whether or not a NITF API is available.  If said component knows about a handy utility to work with NITF Metacards, that'd be great.  If not, that's okay too.  I would expect that the NITF "wrapper" for a Metacard would be a lower priority but when it was developed it would offer nice methods to easily extract the base data, and more generic key/value methods to get at the more extensible TREs which can't be easily integrated into the API.

This approach is based on our experience with the actual Metacard API.  If I were able to go back, I would make the interface much more generic (things like getAttribute and setAttribute) and move the "helper" methods (getTitle, getThumbnail, etc) to a separate JAR.  Requiring alternative implementations of Metacard to implement all the helper boilerplate is really too much to ask when you could simply build one utility class that would work with any Metacard implementation.

Michael

--

[John Main]

Brad,

I'm willing to help out on this project.  I'm looking for a full Java implementation of a NITF reader for our project.  I also would like it to write and I'd be willing to write all the code to write a NITF file to disk.

Regards,

John M. Main

Information Technology Specialist

402nd SMXG/577th SMXS/Flt H

[Brad Hards]

On Wed, 13 Aug 2014 02:24:57 AM John Main wrote:
> I'm willing to help out on this project. I'm looking for a full Java
> implementation of a NITF reader for our project. I also would like it to
> write and I'd be willing to write all the code to write a NITF file to disk.

Thanks for the offer: "accepted" :-)

I've renamed the project (since it obviously won't be a reader any more), and
pushed on getting this into Codice.

For now you can clone from https://github.com/bradh/imaging-nitf but there may
be a need to change your origin (with git remote set-url origin {new url})
once I get it transferred to Codice.

In terms of implementation, I'd like to hide the writer behind the new factory
class (NitfFileFactory), with an API like:

public static boolean writeFile(final NitfFile nitf, final String fileName) {
NitfFileWriter writer = new NitfFileWriter(fileName);
return writer.write(nitf);
}

The exact names don't matter.

Brad