UPDATE: IsoriX 0.9.2 is on CRAN

[Alexandre Courtiol]

Dear friends of IsoriX, the new version of IsoriX is ready!

## In short

This new version should address some key limitations introduced by the last update.
So, if you are using IsoriX 0.9.1, please update ASAP.
Our online documentation (https://bookdown.org/content/782) has also been updated for IsoriX 0.9.2.

## The longer version

Not that long ago, I released the version 0.9.1 of IsoriX which implied a lot of changes underneath the surface.
The goal was for IsoriX to produce the same results as earlier versions, while no longer relying on the packages {sp} and {raster} and relying instead on the more efficient and modern package {terra}.
I had warned you that issues may arise as a consequence of this (necessary) update…
… and that is exactly what happened :-(

I had tested the version 0.9.1 on the examples included in the package and encountered no issue.
However, while trying to update the online documentation (bookdown), I noticed two main problems I had failed to anticipate:
1) plotting did not work properly in some cases.
2) it was no longer possible to save R objects created with IsoriX into binary files.

Solving these two issues proved more difficult but also more interesting than anticipated, so I will share some details here for whoever may be interested.
For the others, it may still be worth reading the section "About backward incompatibilities" below since solving these two issues has forced me to introduce backward incompatible changes in IsoriX (i.e., some R code you wrote may have to be adjusted for it to keep working with IsoriX >= 0.9.1).

### About plotting

The immediate problem that appeared upon building the bookdown was that the ocean masks worked over Europe but put the Americas under water.
I first suspected that I had either rebuilt the ocean mask wrongly, or that I was doing something wrong in the code.
In fact, the problem was more general and had its root much deeper.

It took me a while to understand that and despite some code dissection, what put me on good tracks was the reading of a paper by Paul Murrell on the topic (https://journal.r-project.org/archive/2012/RJ-2012-017/RJ-2012-017.pdf).
The problem was that IsoriX could no longer plot polygons containing holes within them.
At the bottom of plot implementation, R makes a difference between usual polygons with no holes (think of a pancake) and those containing holes (think of a flat donut) which are called paths.
In IsoriX v0.8.3 and earlier, IsoriX had no problem plotting paths, because it relied on a function from {sp} that directly called `grid.path()` from {grid} – a powerful package integrated in base R that is what {ggplot2} and {lattice} actually use to draw things).
But since IsoriX v0.9.1 and beyond no longer use the (antiquated) {sp} package, the functionality had gone.

I discussed the problem with the maintainers from the packages {rasterVis} and {lattice}, and we decided to solve the problem at its source: within {lattice} itself.
I am lucky they agreed since {lattice} is part of the very short list of "recommended" packages provided by CRAN and those packages don't evolve much.
We thus worked together on adapting {lattice} so that it could use `grid.path()`, we then worked on adapting {rasterVis} to exploit those changes made in {lattice}, and I then worked on adapting IsoriX to exploit these changes as well.
So thanks to the drawing of the Americas by Isorix, and thanks to Deepayan Sarkar (maintainer of {lattice}) and Oscar Perpiñán (Lamigueiro) (maintainer of {rasterVis}), all users from {lattice} can now draw holey shapes, all users of {rasterVis} can add masks with holes on top of their rasters, and you, IsoriX folks, can –again– hide isoscape values behind masks.

### About saving

The second (big) problem I faced was that IsoriX version 0.9.1 could no longer save and read binary files from the disk.
One could live without that, but it is very convenient for workflows that take time to run and develop (as our bookdown) as it allows you not to lose objects created with IsoriX after restarting R.
Binary files produced by R generally have one of the following extension: *.rds, *.rda, and *.Rdata.
They are typically produced by the base R functions `save()`, `saveRDS()` and `save.image()`.
In IsoriX, it used to be possible to save isoscape, assignment or other objects by simply using those functions.
That did not work for very large rasters on a computer with limited memory, but otherwise that worked just fine (at least for me and no one else ever complained).

With spatial objects created with {terra} saving R binary using base R functions does not work.
The function runs, but when you try loading the saved objects into R, it won't work.
The reason is that the R objects created by {terra} are only metadata pointing to an object stored outside R.
The base R functions save the metadata, but not the object outside R, which after a short while becomes overwritten by other things (error messages will mention "pointer").

The maintainer of {terra} and other maintainers of alternative spatial packages recommend to directly save files into Geotiff files or into other spatial formats.
That is a fine solution for dealing with one spatial object at a time, but that approach would become very cumbersome with IsoriX because each object can contain various spatial objects within it.
For example, when performing an assignment, the object created contains two rasters and up to three lists of spatial points.
One would thus need to dissect these objects to save them and then rebuild them after reloading them.
Certainly possible, but annoying at best.

After discussing the issue with Roger Bivand (maintainer of {terra}), he remarked that he had actually implemented a method for `saveRDS()` in his package that works for saving directly R objects created with {terra} as binary.
What Roger Bivand did was to turn base R `saveRDS()` into a generic and write special methods for {terra} objects.

[nb: generics are weird functions that do nothing but to identify what other functions need to be called internally to do the job based on the signature of the object fed to the generic.

The functions doing the job are called methods.
What a generic does is thus to dispatch the object to the right method… or to dispatch the right method to the object… I am not fully familiar with the jargon].

The idea is to bring the spatial object within R (called wrapping) before saving the binary file; and to unwrap that object after reloading the object into R from the binary file.

I thus followed this logic by implementing methods for `saveRDS()` specific to IsoriX.

Coding the methods as such was straightforward.
The difficulty was how to declare the methods so that `saveRDS()` directly recognise when an object has been created with IsoriX, or with {terra}, or with any other package, or with base R.
IsoriX already contains various generics and methods, but those interact within the same package and they are all coded using the so-called S3 object system (which I know well and which is intrinsically simple).
On the other hand, here I had to write methods for IsoriX S3 objects that must be dispatched by generics belonging to another package {terra} which uses the more complicated object system (S4).
The combination is not unusual by any means but certainly far outside my comfort zone (don't forget unlike all guys mentioned above, I am a biologist with no formal training in programming).
How to write hybrid methods for S3/S4 is documented across various long documents with esoteric examples in R but while I am a huge fan of John M. Chambers' writing about R, those documentation pages and the equivalent chapters in his otherwise delightful books are just incomprehensible.

[For the little story and taking shortcuts: John M. Chambers co-created the program S, the inspiration behind R.
When others started to create R to mimic S but with a less restrictive licence and compatible with MacOS, S was at version 3.
By the time R was off the ground John Chambers had created S version 4 and revised how objects work there.
Those methods were thus ported to R.
John Chambers has since lobbied a lot for people to use S4 rather than S3 in R, to the point that when writing about S4 he often does not mention he does as he implies that is the one way to go.
Yet, if S4 is more rigorous and useful for complex programming, it is more complex and seems like an overkill for most packages (notable exceptions are Matrix and packages for genomics hosted by Bioconductor).
A working group is currently working on S7 (and yes the number 5 and 6 were already taken; a system called refclasses existing in base R is nicknamed R5 and a package R6 is a great system for encapsulated object oriented programming used by many packages produced by Posit formerly known as RStudio).
But I am digressing…]

I found packages that did manage that, but they did not use {Roxygen2} which I rely on to write the documentation and declare and export objects (including methods).
So that did not help me much.
I thus went through long sessions of trials and errors to tackle this S3/S4 mess.
Early attempts were working until I would try to load {terra}, which would then overwrite IsoriX's methods.
Or things would work, until one would want to save something that is neither produced by {terra} or IsoriX.
In the end, after much debacle, I am pleased to report that it seems to be working in all complex situations I envisioned.

I hope you will not prove otherwise, but do let me know if you do.

### About other things

While battling my R demons, I also realised other imperfections in the code which I fixed.
In particular, I discovered that `terra::aggregate()` behaves differently than `raster::aggregate()`: when aggregating raster cells, the former can produce NAs unless instructed not to, while the latter does not.
This was a useful bug, because it also revealed that the presence of NAs in any of the predictors used in the isoscape models created various problems.

I also encountered a weird problem affecting only development versions between 0.9.1 and 0.9.2 that made predictions go wrong every 100,000 predictions (it was a coercion issue: `as.character(99999) == "99999"` and `as.character(100001) == "100001"` are `TRUE` but `as.character(100000) == "100000"` is `FALSE` since `as.character(100000)` returns "1e+05").

Other things are details in the NEWS file, but I already wrote too much.

### About backward incompatibilities

In the last email introducing IsoriX version 0.9.1, I had warned that the new version may introduce some backward incompatibilities.
I can now tell a little more about that.
Developers always try to limit backward incompatibilities since users don't like those, but there is a hard tradeoff between keeping things as they were while allowing the software to develop in an ecosystem of other packages that also develop.
The good thing is that the new backward incompatibilities only apply at advanced usage of IsoriX.
I think there should be only one true backward incompatibility: one can no longer save and load objects with `save()` and `load()`.
You must use `saveRDS()` and `readRDS()` instead.
Beware that when using `readRDS()`, you need to assign the object that is read (i.e., use `<-` to name and store what you read; see `?readRDS` for details).

If plotting IsoriX objects without using IsoriX functions, you will also have to adapt your code to use {lattice} rather than {sp}.

For plotting, the code previously used to manually add masks was:
`layer(sp.polygons(CountryBorders, col = "white"))` and now it should be:
`layer(lpolygon(CountryBorders, border = "white"))`.

### About the future

All of this means I have not tackled any of the other open issues in IsoriX…
I have reorganised the next goals into categories using milestones in GitHub (https://github.com/courtiol/IsoriX/milestones).
To let me know of your priorities, please visit the issues within the milestones and put thumbs up to those that matter to you.
Also, if you really want something to be solved quickly, please let me know and we will discuss it.

All the best,

Alex

[Leonard Wassenaar]

Alex - it's great to see you are so committed to IsoRix's development and continual improvement. This is rare, as it is mainly benefitting others' work. Well done!  Cheers, Len

--
You received this message because you are subscribed to the Google Groups "IsoriX" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isorix+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isorix/838cc0f6-d86a-4ec2-a942-5a5be6ac2e84n%40googlegroups.com.