Re: [OT] identification of polar granules.

[Ivan Shmakov]

>>>>> James Kuyper <james...@verizon.net> writes:
>>>>> On 05/16/2013 04:36 PM, Ivan Shmakov wrote:

[Cross-posting to news:sci.geo.eos, for it seems like a much
more appropriate group for MODIS-related discussions.]

[...]

>> FWIW, most of the MODIS PGEs' sources I've ever had to deal with
>> (though those were generally L2 and L3; both C4 and C5) weren't
>> anything like a good example of C coding, either.

> Most of that code was written by scientists who know a bit about
> programming.

It's the same at this side of the globe. Seems like a truly
international trait, indeed!

> Contrary to my original plans for my career, I've ended up being
> primarily a programmer who knows a lot about science, so I have some
> idea what you're talking about.

While having similar plans, I think I've by now ended up knowing
way too little about way too much. (Why, among my current plans
are a Web interface to DNS zones, an Ext2+ inspection and backup
tool, and some bits of ARM Cortex M3 MCUs programming.)

> I'm responsible for the following:

> L0: PGE00, PGE95, PGE96, PGE97
> L1: PGE01
> L2: PGE60

> and I'm currently finishing off the C6 changes to those programs.

Curiously, are there any plans to release C6 PGEs as free
software? For sure, I'd feel much better being able to post
diffs (and I've had a few) to Usenet (or mailing lists) than
having to keep them for myself.

> If you've noticed any poor C coding practices in those PGEs, I'd
> appreciate hearing about it.

As for L0 and L1, ISTR that we've ended up using the precompiled
versions that come with SeaDAS, and I don't seem to recall any
issues with them. (And I don't work at that lab anymore.)

> Most of that code was originally written by someone else (usually
> people who knew more about Fortran than about C, and more about
> science than about Fortran). It's also subject to some
> less-than-ideal coding standards and guidelines.

ACK.

--
FSF associate member #7257

[James Kuyper]

On 05/21/2013 08:10 AM, Ivan Shmakov wrote:
>>>>>> James Kuyper <james...@verizon.net> writes:
>>>>>> On 05/16/2013 04:36 PM, Ivan Shmakov wrote:
>
> [Cross-posting to news:sci.geo.eos, for it seems like a much
> more appropriate group for MODIS-related discussions.]
>
> [...]
>
> >> FWIW, most of the MODIS PGEs' sources I've ever had to deal with
> >> (though those were generally L2 and L3; both C4 and C5) weren't
> >> anything like a good example of C coding, either.
>
> > Most of that code was written by scientists who know a bit about
> > programming.
>
> It's the same at this side of the globe. Seems like a truly
> international trait, indeed!

...

> > I'm responsible for the following:
>
> > L0: PGE00, PGE95, PGE96, PGE97
> > L1: PGE01

I forgot to mention PGE92 and PGE93.

> > L2: PGE60
>
> > and I'm currently finishing off the C6 changes to those programs.
>
> Curiously, are there any plans to release C6 PGEs as free
> software? For sure, I'd feel much better being able to post
> diffs (and I've had a few) to Usenet (or mailing lists) than
> having to keep them for myself.

Collection 6 versions of the MODIS software are already freely available at
<https://modaps.nascom.nasa.gov:9500/software/> for M-API, the SDST
Toolkit, PGE01, PGE02, PGE03, and PGE93. That's a password protected
site, you'll first have to set up an account by registering at
<https://modaps.nascom.nasa.gov:9500>. That's just a formality - all
they need is a faxed copy of the NASA Software Usage Agreement with your
signature on it before they'll let you have the software.

The currently publicly released C6 code is mostly mine (PGE03 is the
only part that wasn't - I was still responsible for PGE02 when the C6
code was being developed). That's probably because all of the other PGEs
depend upon the L0 and L1 PGEs, so I had a lot of pressure to complete
my C6 updates before the other groups. I'm not sure when the other C6
PGEs will become publicly available, but there's no intention to make C6
any less freely available than C5.

> > If you've noticed any poor C coding practices in those PGEs, I'd
> > appreciate hearing about it.
>
> As for L0 and L1, ISTR that we've ended up using the precompiled
> versions that come with SeaDAS, and I don't seem to recall any
> issues with them. (And I don't work at that lab anymore.)

Most of that code was originally created by taking much older versions
of the programs I'm responsible for, and heavily modifying them for the
needs of SeaDAS. They did not have enough resources to spare to keep
their version in sync with the changes I've been making to mine. They
were several years behind me the last time I checked (which was several
years ago).
--
James Kuyper

[Ivan Shmakov]

>>>>> James Kuyper <james...@verizon.net> writes:
>>>>> On 05/21/2013 08:10 AM, Ivan Shmakov wrote:

[...]

>> Curiously, are there any plans to release C6 PGEs as free software?
>> For sure, I'd feel much better being able to post diffs (and I've
>> had a few) to Usenet (or mailing lists) than having to keep them for
>> myself.

> Collection 6 versions of the MODIS software are already freely
> available at <https://modaps.nascom.nasa.gov:9500/software/> for
> M-API, the SDST Toolkit, PGE01, PGE02, PGE03, and PGE93. That's a
> password protected site, you'll first have to set up an account by
> registering at <https://modaps.nascom.nasa.gov:9500>. That's just a
> formality - all they need is a faxed copy of the NASA Software Usage
> Agreement with your signature on it before they'll let you have the
> software.

I'm glad to know that the code will still be available free of
charge, but here, I'm concerned with liberty, not price.

For a now-hobbyist I am, especially inconvenient would be the
section 9.iv of the User Agreement, which reads:

[...] Recipient shall provide a written, semiannual report to
Goddard identifying all further Recipients as required by the
Goddard representative designated hereunder.

Given that it's customary to use a context-bearing diff (either
-c or -u diff(1) options) when publishing patches, it's expected
that the change as published would include a portion of the code
covered by the agreement. And as the agreement effectively
disallows for these portions to be distributed to unidentified
parties, the usual means of patch distribution (as in: putting
one on a public Web site, or posting to a newsgroup) instantly
do not seem appropriate any longer.

Surely, it's possible that the publication of such patches may
be covered by "fair use" in US, and the equivalent provisions in
other jurisdictions, but, well, dealing with /two/ distinct
jurisdictions (the US one, and the one at the place of residence
of the interested party) looks somewhat impractical for an
individual, or even a small lab. (Unless one specializes in
software distribution, that is.) A hurdle a conventional free
software license (even if copyleft) would've helped to avoid.

> The currently publicly released C6 code is mostly mine (PGE03 is the
> only part that wasn't - I was still responsible for PGE02 when the C6
> code was being developed). That's probably because all of the other
> PGEs depend upon the L0 and L1 PGEs, so I had a lot of pressure to
> complete my C6 updates before the other groups. I'm not sure when
> the other C6 PGEs will become publicly available, but there's no
> intention to make C6 any less freely available than C5.

ACK, thanks for the information.

BTW, do I understand it correctly that C6 output can be fed into
the later steps of a C5 processing chain? (ISTR that C4 and C5
weren't all that compatible.)

[...]

>> As for L0 and L1, ISTR that we've ended up using the precompiled
>> versions that come with SeaDAS, and I don't seem to recall any
>> issues with them. (And I don't work at that lab anymore.)

> Most of that code was originally created by taking much older
> versions of the programs I'm responsible for, and heavily modifying
> them for the needs of SeaDAS. They did not have enough resources to
> spare to keep their version in sync with the changes I've been making
> to mine. They were several years behind me the last time I checked
> (which was several years ago).

Strangely enough, ISTR that in c. 2009, when I've been reviewing
the changes between the PGEs (as were available from MODAPS back
then) and SeaDAS (one of the latest versions, IIRC), the changes
seemed fairly marginal. And neither I seem to recall any "heavy
modifications" on the SeaDAS side.

I guess we're interested in mod_pr01, mod_pr03 (also included in
PGE01) and mod_pr02 (PGE02) specifically (as well as their
respective wrapper Shell scripts), out of all the sheer variety
of code SeaDAS comes with. The rest of the processing chain was
built from sources as available from MODAPS.

[James Kuyper]

Ah - so that's what you meant with your comment about diffs! I'm not a
lawyer; I'm not sure whether your interpretation is correct. However, if
the diffs you want to publish involve corrections to code I'm
responsible for, I'd appreciate it if you would at least send a copy of
them to me, so I can decide whether I want to incorporate them in future
versions. If so, I'll give you proper credit in the check-in comments.

...

> BTW, do I understand it correctly that C6 output can be fed into
> the later steps of a C5 processing chain? (ISTR that C4 and C5
> weren't all that compatible.)

We didn't change the format of any existing feature of the product
files, we just added new features, and improved the calculations used to
determine the actual contents of the files. Since HDF allows you to
access individual components of a file without needing to know or care
about any other components in the same file, I would have given you an
unqualified "Yes" to that question, if it had not been for one bizarre
problem that was brought to my attention. A downstream program checked
to see whether our products had any HDF-EOS dimension maps, and rejected
the product file as invalid if it had any. Well, one of our C6 changes
was to add high-resolution data to the file, which means we needed to
add a dimension map indicating how the high-resolution data was related
spatially to the low-resolution data. The person currently responsible
for that code was a newbie who had no idea what the validity test even
meant, much less why it was being performed. Budget cuts!

I've learned something on this project that would have seemed
counter-intuitive to me when I was fresh out of grad school: don't
perform input validity tests on features of the input that don't
actually matter to your code - all that those tests do is make it more
difficult to reuse your code in a context where those validity tests are
no longer relevant, and such a context will usually come up, sooner or
later.

...

> > Most of that code was originally created by taking much older
> > versions of the programs I'm responsible for, and heavily modifying
> > them for the needs of SeaDAS. They did not have enough resources to
> > spare to keep their version in sync with the changes I've been making
> > to mine. They were several years behind me the last time I checked
> > (which was several years ago).
>
> Strangely enough, ISTR that in c. 2009, when I've been reviewing
> the changes between the PGEs (as were available from MODAPS back
> then) and SeaDAS (one of the latest versions, IIRC), the changes
> seemed fairly marginal. And neither I seem to recall any "heavy
> modifications" on the SeaDAS side.

I don't pay much attention to SeaDAS; the last two times I checked were
in 2007 and 2010; each time they were running a version of our code that
was more than three years out of date.

Several organizations distribute modified versions of our code; I may be
remembering a different distributor. The modifications I'm thinking
about mainly involved dropping the use of a third-party library that our
code is required to use; they replaced calls to functions in that
library with code intended to serve a similar purpose (except when the
original code did something they had no interest in, in which case they
just commented it out). It looked to me like keeping their version
synchronized with ours would have been a maintenance nightmare, which
would explain why they were 3 years behind us. Had it been my
responsibility, I would have simplified that nightmare by creating an
emulated version of that third party library - but possibly that would
have run into intellectual property rights issues.

[glen herrmannsfeldt]

In comp.lang.c James Kuyper <james...@verizon.net> wrote:
> On 05/30/2013 03:44 PM, Ivan Shmakov wrote:

(snip, someone wrote)

>> >> Curiously, are there any plans to release C6 PGEs as free software?
>> >> For sure, I'd feel much better being able to post diffs (and I've
>> >> had a few) to Usenet (or mailing lists) than having to keep them for
>> >> myself.

(snip)

>> [...] Recipient shall provide a written, semiannual report to
>> Goddard identifying all further Recipients as required by the
>> Goddard representative designated hereunder.

>> Given that it's customary to use a context-bearing diff (either
>> -c or -u diff(1) options) when publishing patches, it's expected
>> that the change as published would include a portion of the code
>> covered by the agreement. And as the agreement effectively
>> disallows for these portions to be distributed to unidentified
>> parties, the usual means of patch distribution (as in: putting
>> one on a public Web site, or posting to a newsgroup) instantly
>> do not seem appropriate any longer.

> Ah - so that's what you meant with your comment about diffs! I'm not a
> lawyer; I'm not sure whether your interpretation is correct. However,
> if the diffs you want to publish involve corrections to code I'm
> responsible for, I'd appreciate it if you would at least send a copy of
> them to me, so I can decide whether I want to incorporate them in future
> versions. If so, I'll give you proper credit in the check-in comments.

Interesting question. I would think that most should be within fair
use, but maybe not so obvious.

Consider instead of software the copyright on a fictional story book.

Now, say you want to publish one sentence for some reason, maybe in
similar context to a diff. (The author could have said ... instead of.)

In most cases, I would think that one line of a story book would
not be unique enough to claim infringement, but maybe not in all cases.

(Try to claim copyright infingement on the C statement i=0; without
any context.)

But maybe some C statements, and maybe some fictional story lines,
could be unique enough.

(snip)

> We didn't change the format of any existing feature of the product
> files, we just added new features, and improved the calculations used to
> determine the actual contents of the files. Since HDF allows you to
> access individual components of a file without needing to know or care
> about any other components in the same file, I would have given you an
> unqualified "Yes" to that question, if it had not been for one bizarre
> problem that was brought to my attention. A downstream program checked
> to see whether our products had any HDF-EOS dimension maps, and rejected
> the product file as invalid if it had any. Well, one of our C6 changes
> was to add high-resolution data to the file, which means we needed to
> add a dimension map indicating how the high-resolution data was related
> spatially to the low-resolution data. The person currently responsible
> for that code was a newbie who had no idea what the validity test even
> meant, much less why it was being performed. Budget cuts!

The XML standard requires readers to ignore any elements or attributes
which they don't need. Others have noticed the problem.

> I've learned something on this project that would have seemed
> counter-intuitive to me when I was fresh out of grad school: don't
> perform input validity tests on features of the input that don't
> actually matter to your code - all that those tests do is make it more
> difficult to reuse your code in a context where those validity tests are
> no longer relevant, and such a context will usually come up, sooner or
> later.

Not always so easy with an unstructured file, but with tags it is
often possible to ignore tags that you don't need.

-- glen

[Ivan Shmakov]

>>>>> James Kuyper <james...@verizon.net> writes:
>>>>> On 05/30/2013 03:44 PM, Ivan Shmakov wrote:

[...]

>> Given that it's customary to use a context-bearing diff (either -c
>> or -u diff(1) options) when publishing patches, it's expected that
>> the change as published would include a portion of the code covered
>> by the agreement. And as the agreement effectively disallows for
>> these portions to be distributed to unidentified parties, the usual
>> means of patch distribution (as in: putting one on a public Web
>> site, or posting to a newsgroup) instantly do not seem appropriate
>> any longer.

> Ah - so that's what you meant with your comment about diffs! I'm not
> a lawyer; I'm not sure whether your interpretation is correct.

Neither am I... and neither am I. I guess that it may fall
under fair use, but as I'm not a layer, I'd certainly appreciate
for the license to clarify this point. Like the GNU GPLs do.

Then, however, every version of GNU GPL allows for "unaccounted"
distribution of the works covered. And I'm unsure if it would
even be possible to devise a condition which would allow for the
patches to be distributed freely, yet forbidding the same for
any "substantial" portion of the code.

> However, if the diffs you want to publish involve corrections to code
> I'm responsible for, I'd appreciate it if you would at least send a
> copy of them to me, so I can decide whether I want to incorporate
> them in future versions. If so, I'll give you proper credit in the
> check-in comments.

That's certainly an option.

However, I am (or at least was) interested chiefly in the latter
stages of the processing. Not to mention that some changes
were, well, not unarguable improvements.

For instance, having found that there're /two/ versions of the
ancillary data format used by mod_pr07 (unless I'm confusing
them, -- one for the newer ancillary data for MODIS/Terra, and
the other for the older, MODIS/Aqua, one) -- and thus /two/
versions of mod_pr07 itself, I've made quite substantial changes
to the loading routine so that it'd handle both formats.

Also, I've replaced the pure-Make build systems of several PGEs
with ones based on GNU Autotools, mainly as this matches my
habits better.

I still believe that these changes might have be of some use to
the community, but I'm unsure if I'd like to go through all the
hurdle of asking everyone interested to sign the license first.

One another way to go would be to open a "community section" at
MODAPS, for the registered users to exchange their patches, etc.

>> BTW, do I understand it correctly that C6 output can be fed into the
>> later steps of a C5 processing chain? (ISTR that C4 and C5 weren't
>> all that compatible.)

> We didn't change the format of any existing feature of the product
> files, we just added new features, and improved the calculations used
> to determine the actual contents of the files. Since HDF allows you
> to access individual components of a file without needing to know or
> care about any other components in the same file, I would have given
> you an unqualified "Yes" to that question, if it had not been for one
> bizarre problem that was brought to my attention.

[...]

ACK, thanks!

> I've learned something on this project that would have seemed
> counter-intuitive to me when I was fresh out of grad school: don't
> perform input validity tests on features of the input that don't
> actually matter to your code - all that those tests do is make it
> more difficult to reuse your code in a context where those validity
> tests are no longer relevant, and such a context will usually come
> up, sooner or later.

Yes.

[...]

>> Strangely enough, ISTR that in c. 2009, when I've been reviewing the
>> changes between the PGEs (as were available from MODAPS back then)
>> and SeaDAS (one of the latest versions, IIRC), the changes seemed
>> fairly marginal. And neither I seem to recall any "heavy
>> modifications" on the SeaDAS side.

> I don't pay much attention to SeaDAS; the last two times I checked
> were in 2007 and 2010; each time they were running a version of our
> code that was more than three years out of date.

> Several organizations distribute modified versions of our code; I may
> be remembering a different distributor. The modifications I'm
> thinking about mainly involved dropping the use of a third-party
> library that our code is required to use; they replaced calls to
> functions in that library with code intended to serve a similar
> purpose (except when the original code did something they had no
> interest in, in which case they just commented it out).

If the third-party library means SDP Toolkit specifically, ISTR
that they rather included its necessary parts into the code they
distribute. (Or I may be confusing them with IMAPP...)

> It looked to me like keeping their version synchronized with ours
> would have been a maintenance nightmare, which would explain why they
> were 3 years behind us. Had it been my responsibility, I would have
> simplified that nightmare by creating an emulated version of that
> third party library - but possibly that would have run into
> intellectual property rights issues.

FWIW, when I've asked the SDP Toolkit maintainers about the
license, they've replied that it's in the public domain. Thus,
unless there was a miscommunication of some kind, I'd rather
advocate using SDP Toolkit -- which seems a surprisingly sound
library -- directly.