iris grib future

[marqh]

The iris-grib module
https://github.com/scitools/iris-grib

was recently removed from Iris, at version 1.10

the management of this library has been problematic, from the point of view of code and of package management

In my view a key cause of this is that the interface between these modules is not well defined and the dependency structure between the libraries is worryingly close to circular.

The work to adopt Dask has highlighted just how challenging it is to maintain these two libraries independently, when in many cases they need to be in lock step.

We are currently evaluating the feasibility of removing iris-grib and reincorporating the code into Iris:
https://github.com/SciTools/iris/projects/4
https://github.com/SciTools/iris/tree/gribiris
https://github.com/SciTools/iris/pull/2476

this work is currently taking place on a feature branch, partially to establish plausibility. A decision on this as the path for Iris 2.0 has not yet been taken

This thread is an opportunity to discuss this approach and bring up potential benefits and costs of this approach

[bjlittle]

I'm +1 for bringing iris-grib back into iris.

The original reasons for porting the grib capability out of iris and into a separate repo (iris-grib) was to de-bloat the iris code base and also to free iris from the release cycle of grib.

This made total sense at the time, but even from early on during the task of porting grib out of iris into iris-grib, it was quite clear (to me) that the two were very, very tightly coupled together.

To be honest, I've not seen the fruition of the major perceived benefits from the port (apart from footprint), and as we're stepping through the task of replacing biggus within iris with dask ... we're hitting several pain points all over again. The separation of iris and iris-grib is really artificial (for me) and on reflection I don't think that we should have done it. That said, hindsight is a really wonderful thing. We live and learn, and we should strive to correct our course in the light of new knowledge and understanding. So for those reasons we shouldn't be scared to retract our initial decision.

I say we retract that decision.

The PR https://github.com/SciTools/iris/pull/2479 is actively targeting bringing iris-grib back into iris ... and I'm reviewing it with the intent of merging it into the iris dask feature branch.

Our plans are for the iris dask feature branch to be the baseline for iris v2.0.0.

If you have any reservations about not bringing iris-grib back into the fold, then please voice your concerns.

[Phil Elson]

I'm entirely against bringing iris-grib back into iris. I've not voiced my concerns so far for a number of reasons, not to mention my desire for the dask work to be integrated into master ASAP.

We are fortunate to be in the position of making an informed decision either way, since the last release of Iris is one where the iris.fileformats.grib module was deprecated. We can choose to revert that deprecation, or to continue down the path of breaking Iris into its separate concerns.

To be completely clear, my long-term aspiration is to reduce the scope of the (core) Iris package and to break it into a number of sub-packages. 

Where possible, these packages should be reusable independent of Iris e.g. cf_units, cartopy, cf_standard_name, biggus/dask, etc.

In practice though, there will need to be packages that are not iris-agnostic. As it happens, there already are many such developments in both the public domain (e.g. eofs, cis), and as private repositories (both in the form of packages, and as user scripts). In order to keep that work moving forward, it is pivotal that we have clear software/API boundaries, and that we document changes to those interfaces clearly and efficiently. To that end, iris-grib has served as a canary for the core changes that we have made recently, and highlight to me that we have some work to do in understanding+documenting those changes (there is no doubt that we've felt the pain - but so too do those other downstream packages).

I'm certainly not saying that splitting Iris into key components is trivial - the mono-repo concept is well documented and loved by many for good reason. Particularly:

• A single place to report bugs
• Shared infrastructure (e.g. travis setup)
• A guarantee that all components work with one another (but absolutely does not guarantee that versions can be mixed and matched)

However, the benefits of splitting Iris are several-fold:

• In general, we have the ability to provide stability to Iris-core, but extend/enhance its functionality, even after a version has been released. 
• Since biggus has recently gained experimental dask capabilities, technically provided a sufficiently new version of biggus is available any version of Iris >=1.7 could parallelise its lazy operations using dask/distributed. Iris 1.7 was released in April 2015, but dask wasn't a mainstream package until after that! To be clear: I'm very much supportive of Iris-core learning to talk dask, rather than biggus.
• With a small amount of effort, it is trivial to make iris-grib work with eccodes (rather than GRIBAPI). With a sufficiently new version of iris-grib, it would be possible for iris >= 1.10 to load grib files using the latest decoder (and even on python 3!). No new Iris release will be needed.
• It is possible to run the iris-grib tests in a fraction of the time to run the full iris test suite, and without the need for installing "optional" dependencies such as matplotlib.
• It is entirely possible to separate the work of updating the core library, without needing to immediately update its dependants (e.g. iris-grib). Making the development much less of a "big-bang" approach.

  

The message has probably gone on for far too long, so I'll stop there. Pertinent changes/pull requests:

• iris-grib making use of the new dask functionality in iris
  
• Entirely removing iris.fileformats.grib from iris (it has already been deprecated)
• biggus gains experimental dask functionality (read: not dask.array)

We do need to make an informed decision on this in fairly short order, and given @bjlittle and I have set our stalls in different places, I suggest we get together and talk it over in a google hangout. I've set up a doodle poll to get the best possible time (though I accept that it won't be able to cover everybody who is interested). If you'd like to be involved, please add yourself to  https://beta.doodle.com/poll/yaxm4pqfkus96sm2.

Cheers,

Phil

[Phil Elson]

Votes are in...

In order to give as much lead time as possible I've closed the poll and selected Friday 7th July at 1100 BST. (https://doodle.com/poll/yaxm4pqfkus96sm2). The one big potential issue is that I don't currently have a response from Bill, and I'm aware he has a few prior arrangements on Friday - I'll know by Weds whether we need to amend the date (likely it would be postponed until early next week if so).

In terms of how we move this forward, I'm keen that we have as many perspectives as possible and hope we can come to a consensus in the meeting. In the absence of a consensus, and no BDFL, as the M.O.'s SciTools technical leads Bill and I will weigh up the evidence and come to an agreement.

The Google hangouts link will be https://hangouts.google.com/call/mihrcjiybzbfvmvz2ientzl774y.

Thanks,

--
You received this message because you are subscribed to the Google Groups "Iris-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scitools-iris-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Phil Elson]

Sorry for the last minute change of schedule. Seems like Bill can't join us at 11 on Friday, so I propose we postpone to early next week. Doodle: https://doodle.com/poll/6vemh6v6i6t4rnhm

I'll update here with results/plan.

[Phil Elson]

Thanks for those who responded (again). I've picked the 1300-1400 slot (1PM-2PM BST).

Reminder: https://hangouts.google.com/call/mihrcjiybzbfvmvz2ientzl774y (you can visit that link at anytime to check your equipment works on hangouts).

[Phil Elson]

Please find my notes from the meeting (other attendees, please feel free to add further pertinent discussion points):

Attendees: Andrew Dawson (ajdawson), Mark Hedley (marqh), Bill Little (bjlittle), Phil Elson (pelson)

• iris-grib separation occurred ~1 year ago because:
  
• iris.fileformats.grib had some changes that we were keen to get released, but were slowed down by the iris release cycle (around the time we deprecated the old grib functionality)
• iris had some changes that we were keen to release, but were aware that iris.fileformats.grib needed significant re-working before we could actually cut anything (we were talking about iris 2 for 2016!)
• in short, we wanted to decouple the iris and iris.fileformats.grib release cycles
• iris was managing a number of deprecations within iris.fileformats.grib that were adding significant complexity to the codebase
• having iris-grib as a separate entity allowed us to target our testing matrix better (e.g. specific versions of gribapi, python versions etc.)

• iris-grib re-introduction occurred during the dask development as: 
  
• it was a pragmatic choice to bring the iris-grib code closer to the iris-core code while developing the dask feature
• it meant we didn't have to mirror the feature branches in iris-grib for CI

• Since making the separation we have learnt:
• There is some connective tissue between iris and iris-grib
• iris.save has a lazy import of iris_grib and calls the iris_grib.save_grib2 function
• iris.load has a lazy import of iris_grib and calls the iris_grib. load_cubes function
• There are integration tests within iris that are testing iris_grib

• Observations
• It is noted that iris-grib fundamentally depends upon iris (for the Cube concept)
• conda install iris does not install iris-grib. If it did, there would be some circular dependency headaches (or iris-grib couldn't depend on iris in the conda recipe)
• We haven't made much use of the separate release cycles - iris-grib development hasn't accelerated, nor has iris itself
• iris-grib took significantly longer than was anticipated to pull out of iris. Putting it back into iris was relatively pain free.
• The added risk of taking iris-grib back out of master for iris 2.0 isn't ideal
• pelson: I have the PRs - it was manageable (~2 hours) and consider there to be little risk 
• Smaller packages is generally the preferred direction of travel (e.g. biggus, nc_time_axis, cf_units, cartopy), but iris_grib is different as it is the first one that depends on iris
• bjlittle: iris-grib is about the hardest case, and we should strive to solve it to simplify this for other breakout/satellite packages

• Places for improvement
• iris and iris-grib currently have tight version coupling (we typically need to release one at the same time as the other)
• potential to improve the architecture / reduce the coupling between iris and iris-grib needs to be explored (e.g. entrypoints)

It is worth noting that we didn't come to a unanimous vote, but having considered all of the content we discussed, a vote of 3-1 supported continuing down the path of breaking iris into smaller components wherever possible, starting with iris-grib. I appreciate this isn't everybody's favoured outcome and want to make sure we find the space to focus on the areas that have been highlighted as "Places for improvement".

Thanks to everybody who took the time to feed back thoughts and opinions both offline and as part of this discussion.

[marqh]

Regarding:

 

• iris-grib re-introduction occurred during the dask development as: 
  
• it was a pragmatic choice to bring the iris-grib code closer to the iris-core code while developing the dask feature
• it meant we didn't have to mirror the feature branches in iris-grib for CI

I think the integration of iris.fileformats.grib to assist in development of the dask interface was motivated by concern around the circular dependency between iris and iris-grib and the problems this caused in development and testing.  There are a set of challenging design questions for iris-grib to answer, which were brought into sharp relief by the development of the dask interface; integration made the answers to a set of design questions simple and clear. 
Having nearly completed this work, there is a proposal to adopt these code changes into the iris-grib module instead of keeping them on master.  This re-raises design questions, which I have not seen explicitly addressed.  It is hard to compare the concrete implementation on iris master to an iris-grib proposal which does not address these questions.

Questions include
* iris.load is dependent on iris-grib, iris-grib is dependent on iris, this is a circular dependency. 
 * Are there aspirations to alter this? What are the implications for the iris API?
 * iris-grib is opaque to many users, it is very often installed and in use via the iris API and iris.load as part of a managed installation.  The impression is that GRIB support is a core capability of the iris API. 
 * given the aspirations to cut iris v2.0 soon, I expect that an iris2 that works with iris-grib will not deliver any change to this relationship.  I appreciate conversations are occurring in this space, but they are immature
* how are iris and iris-grib versions handled at irisv2?
 * which versions of iris are compatible with which versions of iris-grib
 * how is this documented for users
 * how is this presented in installation instructions and installation tools:
  * for conda users?
  * for non-conda users?
  * what is the future plan for versioning of two libraries?
   * is this in lock step?
   * is this iris documentation, iris-grib documentation, both?
   * what are the implications of a proposed future plan
* what is the approach for continuous integration testing of iris, which requires iris-grib; and iris-grib, which requires iris?
 * the previous implementation was that:
  * iris master testing only used the latest released version of iris-grib from conda-forge
  * iris-grib master only used iris v1.10 and iris master, directly from github
 * what is the proposed test environment approach for iris and iris-grib for the master of each of these libraries
 * what are the implications of this for managing releases and for having confidence in statements made about compatibility between versions?
* how is ECCodes adoption proposed to be handled?

I feel it is important that questions including these are answered definitively as part of a concrete proposal.  At present there is uncertainty in my mind about what is being signed up to if a package of work to bounce iris.fileformats.grib back out of master is pursued to deliver in time for the iris 2.0 release.  This presents risks of complication and overrun

ECCodes adoption is an important factor, given the deprecation of GRIBAPI.  Independent of decisions on the future of iris.fileformats.grib and iris-grib, I would like to see
https://github.com/SciTools/iris/pull/2607
reviewed and merged onto master (note: some questions have been raised, and commented on, but this does not feel like it is about to be merged).
There are material questions on that PR regarding ECCodes behaviour changes and implications for maintaining iris behaviour that have not been agreed.  I do not want this PR to be left to atrophy and be closed unmerged.
I think this is small but tangible activity that should happen now, and will be useful and necessary work whatever course of action is implemented for iris 2.0

Is someone prepared to assign themself to #2607 and review those changes with a view to merging, in the knowledge that such an act is not prejudging this ongoing discussion?