[R] Change in behaviour of sd()

[Fiona Johnson]

Hi

I have just upgraded from R2.6.0 to R2.7.1 (running on Windows) and a part
of my code that previously ran ok now gives an error. The following is a
simple example to demonstrate my problem.

> a <- array(c(1,2,3,4,5,6,rep(NA,6)),dim=c(6,2))
> apply(a,2,sd,na.rm=T)

In R2.6.0 this gives (which is what I would like)

[1] 1.870829 NA

In R2.7.1 it gives the following error

"Error in var(x, na.rm = na.rm) : no complete element pairs"

As my columns are always either all NA or all numbers, I could get around it
by replacing the NA's with 0's but if someone could shed some light on why
the behaviour has changed in the new version or a better work around it
would be much appreciated. I want to keep the columns of NA's because
ultimately I am plotting the results with contour and the NA's refer to grid
cells not on land where I don't want to have contours.

Thanks for your help

Fiona

_____________

Fiona Johnson
PhD Candidate, UNSW Water Research Centre
School of Civil & Environmental Engineering
The University of New South Wales, Sydney, NSW, 2052, Australia
Email: fiona.johnson (at) student.unsw.edu.au

______________________________________________
R-h...@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

[Rolf Turner]

On 8/07/2008, at 7:38 PM, Fiona Johnson wrote:

> Hi
>
> I have just upgraded from R2.6.0 to R2.7.1 (running on Windows) and
> a part
> of my code that previously ran ok now gives an error. The following
> is a
> simple example to demonstrate my problem.
>
>> a <- array(c(1,2,3,4,5,6,rep(NA,6)),dim=c(6,2))
>> apply(a,2,sd,na.rm=T)
>
> In R2.6.0 this gives (which is what I would like)
>
> [1] 1.870829 NA
>
> In R2.7.1 it gives the following error
>
> "Error in var(x, na.rm = na.rm) : no complete element pairs"
>
> As my columns are always either all NA or all numbers, I could get
> around it
> by replacing the NA's with 0's but if someone could shed some light
> on why
> the behaviour has changed in the new version or a better work
> around it
> would be much appreciated. I want to keep the columns of NA's because
> ultimately I am plotting the results with contour and the NA's
> refer to grid
> cells not on land where I don't want to have contours.

I just scanned through the release announcements (from Peter
Dalgaard) about new
versions of R (R home page --> What's new? --> Archive of important
announcements)
and found nothing about new behaviour for sd/var/cov. So I cannot
contribute
to enlightenment about ``why''. The following function might provide
a suitable
workaround:

my.sd <- function (x, na.rm = FALSE)
{
if (is.matrix(x))
apply(x, 2, my.sd, na.rm = na.rm)
else if (is.vector(x)) {
if(na.rm) x <- x[!is.na(x)]
if(length(x) == 0) return(NA)
sqrt(var(x, na.rm = na.rm))
}
else if (is.data.frame(x))
sapply(x, my.sd, na.rm = na.rm)
else {
x <- as.vector(x)
my.sd(x,na.rm=na.rm)
}
}

It seems to work on your toy example at least. Note that my.sd
(numeric(0)) returns NA,
but my.sd(NULL) throws an error.

cheers,

Rolf Turner

######################################################################
Attention:\ This e-mail message is privileged and confid...{{dropped:9}}

[Henrik Bengtsson]

Hi,

FYI, *the* NEWS file containing updates for all R versions is
available at http://cran.r-project.org/ - see the link 'new features
and bug fixes'. This links to the URL (which I think is rather
stable):

https://svn.r-project.org/R/trunk/NEWS

The NEWS file also comes with your R installation.

The error message is also there for R v2.7.0 (2008-04-22). So, if it
would be documented in the NEWS, it should be between R v2.6.0 and R
v2.7.0. ...and guess what, it is sitting there under "BUG FIXES" for
R v2.7.0:

" o co[rv](use = "complete.obs") now always gives an error if there
are no complete cases: they used to give NA if
method = "pearson" but an error for the other two methods.
(Note that this is pretty arbitrary, but zero-length vectors
always give an error so it is at least consistent.)"

(It was a bit hard to search for, because neither "sd", "var", "cov"
is in that chunk of text).

/Henrik

[Duncan Murdoch]

This is the relevant but not so obvious NEWS entry:

o co[rv](use = "complete.obs") now always gives an error if there
are no complete cases: they used to give NA if
method = "pearson" but an error for the other two methods. (Note
that this is pretty arbitrary, but zero-length vectors always
give an error so it is at least consistent.)

cor(use="pair") used to give diagonal 1 even if the variable
was completely missing for the rank methods but NA for the
Pearson method: it now gives NA in all cases.

(sd calls var, which calls cov internally.)

Duncan Murdoch

[Peter Dalgaard]

Thanks. I was searching for it as well... (BTW, Rolf, I'm not the
oracle, I just copy parts of the NEWS file into release announcements.
The file itself is readily available, as others point out.)

I think this has come up before but I forgot the details.

It _is_ a bit odd that the variance of an empty vector is an error but
that of a one-element vector is NA, and the difference var(NA,na.rm=T)
and var(numeric(0)) is not too clear either.

Also,

> var(NA,use="pair")
[1] NA
> var(NA,use="co")
Error in var(NA, use = "co") : no complete element pairs

--
O__ ---- Peter Dalgaard Øster Farimagsgade 5, Entr.B
c/ /'_ --- Dept. of Biostatistics PO Box 2099, 1014 Cph. K
(*) \(*) -- University of Copenhagen Denmark Ph: (+45) 35327918
~~~~~~~~~~ - (p.dal...@biostat.ku.dk) FAX: (+45) 35327907

[Jim Lemon]

On Tue, 2008-07-08 at 17:38 +1000, Fiona Johnson wrote:
> Hi
>
> I have just upgraded from R2.6.0 to R2.7.1 (running on Windows) and a part
> of my code that previously ran ok now gives an error. The following is a
> simple example to demonstrate my problem.
>
> > a <- array(c(1,2,3,4,5,6,rep(NA,6)),dim=c(6,2))
> > apply(a,2,sd,na.rm=T)
>
> In R2.6.0 this gives (which is what I would like)
>
> [1] 1.870829 NA
>
> In R2.7.1 it gives the following error
>
> "Error in var(x, na.rm = na.rm) : no complete element pairs"
>
> As my columns are always either all NA or all numbers, I could get around it
> by replacing the NA's with 0's but if someone could shed some light on why
> the behaviour has changed in the new version or a better work around it
> would be much appreciated. I want to keep the columns of NA's because
> ultimately I am plotting the results with contour and the NA's refer to grid
> cells not on land where I don't want to have contours.
>

Hi Fiona,
I get the same behavior on Linux. You can get around it like this:

mysd<-function(x,...) if(!all(is.na(x))) sd(x,...)
apply(sd,2,mysd,)

While I haven't looked at the changes for 2.7.1 on this, I think is it
probably to avoid a problem with trying to calculate the covariance of a
NULL object. If I remember, I'll test it tomorrow at work where I have
access to Windows and email you again.

Jim