merging cubes in iris

[Carwyn Pelley]

It is clear that clarification is necessary over the cube merging process in Iris.
A great deal of misunderstanding is common among the user community on what is deemed a compatible cube to merge and what is not.

I propose the following:

1. A verbose keyword that indicates why a merging did not take place.
2. A technical document detailing how the merging of cubes work in Iris. (related iris_dev discussion - "Iris and Cartopy technical documents" - link)
   

[bblay]

Regarding proposal 1:

What might the output look like?

[Carwyn Pelley]

Just perhaps a printout listing of what is different:

example:

Unable to merge cube: cubename with cube: cubename2
    Details:
    Attribute: History
    Dimension Coordinate: Time

Unable to merge cube: cubename3 with cube: cubename4
    Details:
    ....
.........

I'm not really sure, but it would obviously have to be human readable so that this information could be acted upon by both a developer for possible debugging purposes but also by a general user.

[bblay]

Isn't the reason always going to be because there are no differences?

[pp-mo]

On the contrary, as I understand it, it should be something like :  "two cubes need to differ in exactly one (?relevant? ? ) coordinate to be mergable".
If there are no differences, you can't merge.  If there are more than one, you can't either (well you could, in principle, but I think we don't).

Well, I say it is something like that, but we have never had a straightforward definition.
IMHO, this is exactly why we really need a proper description of the process !
  -- without one, we just don't know what is a bug, and what is "expected" behaviour.

[bjlittle]

I completely agree that the cube merge process requires to be as transparent as possible, particularly now that the user base of Iris is growing
and as core developers we should all strive to understand our code base. Otherwise, how can be provide quality support to our users?

As the cube merge process is rather abstract (opposed to magical) I think that in this case a succinct high level technical note would be very helpful
to both users and developers. It should contain enough detail to clarify the context of what cube merge does, rather than how it does it.

I'd be happy to write such a tech note, but only if we all agree it's a sensible approach to take.

I certainly think that some Iris tool support for cube merge would be helpful. I not too sure about baking in a verbose mode to Iris ... I'd rather see a
suite of one or more tools built upon Iris e.g. a merge-diff tool to allow developers/users to understand why a list of cubes don't merge when the expectation
is that they should.

[RHattersley]

On Tuesday, November 6, 2012 4:52:41 PM UTC, bjlittle wrote:

... I think that in this case a succinct high level technical note would be very helpful

to both users and developers. It should contain enough detail to clarify the context of what cube merge does, rather than how it does it.

 

Agreed - that's a very good proposal. And thanks for the offer to write it - I'd certainly support that.

 

I certainly think that some Iris tool support for cube merge would be helpful. I not too sure about baking in a verbose mode to Iris ... I'd rather see a
suite of one or more tools built upon Iris e.g. a merge-diff tool to allow developers/users to understand why a list of cubes don't merge when the expectation
is that they should.

An excellent idea - it gives much more scope for richness and avoids cluttering the merge code path with `print stuff if verbose`.

Richard

[marqh]

This seems like an excellent idea to me too: gets my thumbs up

[bjlittle]

Okay 2x +1 is good enough for me, I'll start to draft up a tech note.

It would be really useful if we could review it as a group to ensure that it's fit for purpose.

Any thoughts on:

• How we should review it? Group feedback is king IMHO, so do we use this forum, GitHub, or some other approach? It would be kinda nice if such a review process was transparent to all who cared.
• Where should this first of possibly many more to come tech notes live? Within the Iris documentation itself i.e. in Sphinx land, or on separately on scitools.org.uk some how/where?

In answer to my own questions I'd vote for the tech note to live within the Iris documentation, this would allow GitHub to be used for the normal group PR comments and change cycle before merging with upstream/master.

-Bill

[pbentley]

+1

If it's of any help, you may find it useful to have a look at the online docs for the NCO tools to see how they go about describing various merge-type operators. The ncrcat tool is a reasonable example. Plus there's the general overview of NCO concatenators and averagers. Apologies if you knew of these already!

Phil

[bblay]

Do we have to type "+1" in this way?
Isn't there proper way to vote for a topic?
There is the   g+1  at the top but i guess that broadcasts it socially :|

My 1 is simultaneously plussed and nonplussed :D

[bjlittle]

Thanks Phil!

I didn't know that, so I'll have a look.

-Bill

[Carwyn Pelley]

+1 to merge-diff tool

Thread of technical documentation already exists please continue this discussion relating to this on that thread as it is definitely does not only relate to merging.
https://groups.google.com/forum/#!topic/scitools-iris-dev/0yEO_pphKEw

[Carwyn Pelley]

I think there is a misunderstanding, the issue is not whether a cube is the same as another cube.  The issue is in analysing what difference or no difference resulted in an unsuccessful merge.
From my experience on support, discovering where a difference is within the cubes as of now follows a brute force method, testing everything.  This is somewhat a lengthy process.

[Carwyn Pelley]

bump!

Summary of issue:
1. No information relating to how the merge process works in Iris.
2. No way to determine other than brute-force of how cubes differ in such a way that merge does not occur.

Summary of discussion from thread:
1. Technical document detailing the way in which the merge process works (thread).
2. merge-diff tool for understanding why what is an automated process "fails" to behave as some users might expect.

Proposal:
Escalate the issue with the creation of a development ticket within github relating to a merge-diff tool for users and developers alike.

Related threads: link, link, link
link

[Niall Robinson]

I agree with all of the above. On a more basic level, I have been using IRIS/Python since mid November, and after reading the documentation, have no idea what merging is! If it does what I presume it does, then its fundamental to using IRIS properly. I have a list of identical cubes apart from their time but they won't merge, giving the error message "*** ValueError: The points array must be strictly monotonic.", which doesn't really help. I presume I should be able to merge this list of 2D cubes into one 3D cube with an extra time dimension?

Thanks
Niall 

[LeonH]

I agree that we need to know why cubes do not merge. I have a cubelist with lat/lon fields at different times that refuse to merge, no idea why.

Could it be because time is a DimCoord of length 1, rather than a scalar coord? (I say this as when I have seen merge working time was a scalar coord). However, Iris documentation does not explain what a scalar coord is or how to define one, so I am stuck. Any ideas?

[Carwyn Pelley]

Merging definitely needs at least some documentation, I'm looking into this now.  In terms of more substantial documentation (technical documentation)/tools I hope this is to come.

With regards to scalar coordinates, this is a cf concept and so is not specifically documented in iris (since iris follows the cf data model):
http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#scalar-coordinate-variables

The key points to scalar coordinates from the above link are:
1. "...variable has an associated coordinate which is single-valued"
2. "there is no associated dimension"

For merging within Iris, currently it will only work with cubes of different scalar coordinates (the dimension coordinates and the rest of the cube must be identical).  This way, you turn your set of cubes which differ by only their scalar coordinate value into a new merged cube with a new dimension coordinate which corresponds to merging of the scalar coordinates of the individual cubes.  You can however get around this restriction by slicing your cube (i.e. your cube of time dimension coordinate length 1, will turn into a cube with scalar coordinate if you slice it).  You can then merge these slices.

[Niall Robinson]

Hello everyone - are there any thoughts on developing cube.is_compatible(othercube) to be a bit more informative? I usually only use it when I know the answer is going to be "False". It would be lovely if it could just print out where it failed. Sometimes its not obvious to the untrained eye that iris would worry about certain things (e.g. time stamp in the history being two mins different - see LeonH's latest post) It would be great just to know its failed at the history comparison