[R] Another quantmod question
Russ Abbott
quantmod deal with stock quotes. I've created an array of the first 5
closing prices from Jan 2007. (Is there a problem that the name is the same
as the variable name? There shouldn't be.)
> close
close
2007-01-03 1416.60
2007-01-04 1418.34
2007-01-05 1409.71
2007-01-08 1412.84
2007-01-09 1412.11
When I try to create a more complex array by adding columns, the names get
fouled up. Here's a simple example.
> cbind(changed.close = close+1, zero = 0, close)
close zero close.1
2007-01-03 1417.60 0 1416.60
2007-01-04 1419.34 0 1418.34
2007-01-05 1410.71 0 1409.71
2007-01-08 1413.84 0 1412.84
2007-01-09 1413.11 0 1412.11
The first column should be called "changed.close", but it's called "close".
The second column has the right name. The third column should be called
"close" but it's called "close.1". Why is that? Am I missing something?
If I change the order of the columns and let close have its original name,
there is still a problem.
> cbind(close, zero = 0, changed.close = close+1)
close zero close.1
2007-01-03 1416.60 0 1417.60
2007-01-04 1418.34 0 1419.34
2007-01-05 1409.71 0 1410.71
2007-01-08 1412.84 0 1413.84
2007-01-09 1412.11 0 1413.11
Now the names on the first two columns are ok, but the third column is still
wrong. Again, why is that? Apparently it's not letting me assign a name to
a column that comes from something that already has a name. Is that the way
it should be?
I don't get that same problem on a simpler example.
> IX <- cbind(I=0, X=(1:3))
> IX
I X
[1,] 0 1
[2,] 0 2
[3,] 0 3
> cbind(Y = 1, Z = IX[, "I"], W = IX[, "X"])
Y Z W
[1,] 1 0 1
[2,] 1 0 2
[3,] 1 0 3
Is this a peculiarity to xts objects?
Thanks.
*-- Russ *
*
*
P.S. Once again I feel frustrated because it's taken me far more time than
it deserves to track down and characterize this problem. I can fix it by
using the names function. But I shouldn't have to do that.
[[alternative HTML version deleted]]
______________________________________________
R-h...@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
David Winsemius
On May 8, 2011, at 3:07 PM, Russ Abbott wrote:
> I'm having troubles with the names of columns.
>
> quantmod deal with stock quotes. I've created an array of the first 5
> closing prices from Jan 2007. (Is there a problem that the name is
> the same
> as the variable name? There shouldn't be.)
>
>> close
>
> close
>
> 2007-01-03 1416.60
>
> 2007-01-04 1418.34
>
> 2007-01-05 1409.71
>
> 2007-01-08 1412.84
>
> 2007-01-09 1412.11
>
>
> When I try to create a more complex array by adding columns, the
> names get
> fouled up. Here's a simple example.
>
>> cbind(changed.close = close+1, zero = 0, close)
I suspect that you are actually using xts objects that you are
incorrectly calling 'array's. If something is puzzling about the
behavior of an R object the first thing to do is see what you are
really dealing with so ... str(object)
If you load the xts package and type ?cbind.xts , you get a help page
for merge.xts.
(In base R I do not know of a way to assign columns the way you
propose within a `merge` call.)
Here is the code for cbind.xts:
> cbind.xts
function (..., all = TRUE, fill = NA, suffixes = NULL)
{
merge.xts(..., all = all, fill = fill, suffixes = suffixes)
}
<environment: namespace:xts>
David Winsemius, MD
West Hartford, CT
Joshua Wiley
On Sun, May 8, 2011 at 12:07 PM, Russ Abbott <russ....@gmail.com> wrote:
> I'm having troubles with the names of columns.
>
> quantmod deal with stock quotes. I've created an array of the first 5
> closing prices from Jan 2007. (Is there a problem that the name is the same
> as the variable name? There shouldn't be.)
>
>> close
>
> close
>
> 2007-01-03 1416.60
>
> 2007-01-04 1418.34
>
> 2007-01-05 1409.71
>
> 2007-01-08 1412.84
>
> 2007-01-09 1412.11
It would be appreciated in the future if you provided the object via
dput() or some such that is easy to paste in.
>
>
> When I try to create a more complex array by adding columns, the names get
> fouled up. Here's a simple example.
>
>> cbind(changed.close = close+1, zero = 0, close)
>
> close zero close.1
>
> 2007-01-03 1417.60 0 1416.60
>
> 2007-01-04 1419.34 0 1418.34
>
> 2007-01-05 1410.71 0 1409.71
>
> 2007-01-08 1413.84 0 1412.84
>
> 2007-01-09 1413.11 0 1412.11
>
>
> The first column should be called "changed.close", but it's called "close".
> The second column has the right name. The third column should be called
> "close" but it's called "close.1". Why is that? Am I missing something?
Yes.
mat <- matrix(1:10, dimnames = list(NULL, "A"))
cbind(X = 11:20, Y = mat + 1)
cbind(X = 11:20, Y = mat[, "A"] + 1)
In particular note that:
class(mat)
class(mat[, "A"])
It is silly to expect to be able to pass a single name for an entire
matrix, while it makes complete sense that that would work for a
vector. The problem with naming the column the same thing as the
object containing it, is our limited human minds get ramfeezled (R
does just fine, as you can see).
Nope, but do check the type of object you are working with---there are
often different methods for different objects so I would not assumet
that all things would work the same.
Cheers,
Josh
>
> Thanks.
>
> *-- Russ *
> *
> *
> P.S. Once again I feel frustrated because it's taken me far more time than
> it deserves to track down and characterize this problem. I can fix it by
> using the names function. But I shouldn't have to do that.
>
> [[alternative HTML version deleted]]
Please post using plain text.
>
> ______________________________________________
> R-h...@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
--
Joshua Wiley
Ph.D. Student, Health Psychology
University of California, Los Angeles
http://www.joshuawiley.com/
Russ Abbott
mat <- matrix(1:10, dimnames = list(NULL, "A"))
cbind(X = 11:20, Y = mat + 1)
cbind(X = 11:20, Y = mat[, "A"] + 1)
In the line, cbind(X = 11:20, Y = mat + 1), it would be nice if an error or
warning message were issued to the effect that the "Y = " part is ignored or
not applicable or something to that effect.
I still don't understand why cbind.xts doesn't do what I expect. If I try
what Josh suggests on my xts example, the result is still confusing.
> cbind.xts(close, A = close[,"close"])
close close.1
2007-01-03 1416.60 1416.60
2007-01-04 1418.34 1418.34
2007-01-05 1409.71 1409.71
2007-01-08 1412.84 1412.84
2007-01-09 1412.11 1412.11
If this helps:
*
*
*
> dput(close)
*
*
structure(c(1416.6, 1418.34, 1409.71, 1412.84, 1412.11), .indexCLASS =
"Date", .indexTZ = structure("", .Names = "TZ"), src = "yahoo",
updated = structure(1304875186.625, class = c("POSIXct",
*
*
"POSIXt")), class = c("xts", "zoo"), index = structure(c(1167811200,
*
*
1167897600, 1167984000, 1168243200, 1168329600), tzone = structure("",
.Names = "TZ"), tclass = "Date"), .Dim = c(5L,
*
*
1L), .Dimnames = list(NULL, "close"))
*
*
*
I was also unable to convert Josh's matrix example to xts.
> as.xts(mat, order.by = rownames())
Error in dimnames(x) : 'x' is missing
*
*
*The order.by parameter is required, but I don't understand the error
message and haven't been able to figure out how to fix it. If this helps:*
*
*
*
> dput(mat)structure(1:10, .Dim = c(10L, 1L), .Dimnames = list(NULL, "A"))
*
*
*
*This suggests that mat already has defined dimnames. So why does as.xts
give an error message about dimnames? I was unable to figure out the
problem by looking at any of the help files. In particular the help file
for **as.xts.methods didn't seem to offer any clarifications--at least none
that I could understand.*
*
*
*
*
*-- Russ *
On Sun, May 8, 2011 at 12:26 PM, Joshua Wiley <jwiley...@gmail.com>wrote:
> class(mat)
> class(mat[, "A"])
>
[[alternative HTML version deleted]]
Jeff Ryan
Colnames don't get rewritten if they already exist. The reason is due to performance and how cbind is written at the R level.
It isn't perfect per se, but the complexity and variety of dispatch that can take place for cbind in R, as it isn't a generic, is quite challenging to get to behave as one may hope. After years of trying I'd say it is nearly impossible to do what you want without causing horrible memory issues on non trivial objects they are use in production systems **using** xts on objects with billions of rows. Your simple case that has a simple workaround would cost everyone using in the other 99.999% of cases to pay a recurring cost that isn't tolerable.
If this is frustrating to you you should stop using the class.
Jeff
Jeffrey Ryan | Founder | jeffre...@lemnica.com
> -- Russ
Russ Abbott
The xts class has some very nice features, and you have done a valuable
service in developing it.
My primary frustration is how difficult it seems to be to find out what went
wrong when my code doesn't work. I've been writing quite sophisticated code
for a fairly long time. It's not that I'm new to software development.
The column name rule is a good example. I'm willing to live with the rule
that column names are not changed for efficiency sake. What's difficult for
me is that I never saw that rule anywhere before. Of course, I'm not an R
expect. I've been using it for only a couple of months. But still, I would
have expected to run into a rule like that.
Worse, since the rule is in conflict with the explicit intent of cbind--one
can name columns when using cbind; in fact the examples illustrate how to do
it--it would really be nice of cbind would issue a warning when one attempts
to rename a column in violation of that rule. Instead, cbind is silent,
giving no hint about what went wrong.
It's those sorts of things that have caused me much frustration. And it's
these sorts of things that seem pervasive in R. One never knows what one is
dealing with. Did something not work because there is a special case rule
that I haven't heard of? Did it not work because a special convenience was
programmed into a function in a way that conflicted with normal use? Since
these sorts of things seem to come up so often, I find myself feeling that
there is no good way to track down problems, which leads to a sense of
helplessness and confusion. That's not what one wants in a programming
language.
-- Russ
> *-- Russ *
> *
> *
Joshua Ulrich
On May 8, 2011 6:29 PM, "Russ Abbott" <russ....@gmail.com> wrote:
>
> Hi Jeff,
>
> The xts class has some very nice features, and you have done a valuable
> service in developing it.
>
> My primary frustration is how difficult it seems to be to find out what went
> wrong when my code doesn't work. I've been writing quite sophisticated code
> for a fairly long time. It's not that I'm new to software development.
>
> The column name rule is a good example. I'm willing to live with the rule
> that column names are not changed for efficiency sake. What's difficult for
> me is that I never saw that rule anywhere before. Of course, I'm not an R
> expect. I've been using it for only a couple of months. But still, I would
> have expected to run into a rule like that.
>
> Worse, since the rule is in conflict with the explicit intent of cbind--one
> can name columns when using cbind; in fact the examples illustrate how to do
> it--it would really be nice of cbind would issue a warning when one attempts
> to rename a column in violation of that rule. Instead, cbind is silent,
> giving no hint about what went wrong.
>
Naming columns is not the explicit intent of cbind. The explicit
intent is to combine objects by columns. Please don't overstate the
case.
While the examples for the generic show naming columns, neither
?cbind.zoo or ?cbind.xts have such examples. That's a hint.
> It's those sorts of things that have caused me much frustration. And it's
> these sorts of things that seem pervasive in R. One never knows what one is
> dealing with. Did something not work because there is a special case rule
> that I haven't heard of? Did it not work because a special convenience was
> programmed into a function in a way that conflicted with normal use? Since
> these sorts of things seem to come up so often, I find myself feeling that
> there is no good way to track down problems, which leads to a sense of
> helplessness and confusion. That's not what one wants in a programming
> language.
>
If that's not what one wants, one can always write their own
programming language.
Seriously, it seems like you want to rant more than understand what's
going on. You have the R and xts help pages and the source code. The
"Note" section of help(cbind) tells you that the method dispatch is
different. It even tells you what R source file to look at to see how
dispatching is done. Compare the relevant source files from
base::cbind and xts::cbind.xts, look at the "R Language Definition"
manual to see how method dispatch is normally done.
But you've been writing quite sophisticated code for a fairly long
time, so I'm not telling you anything you don't know... you just don't
think you should have to do the legwork.
> -- Russ
>
>
--
Joshua Ulrich | FOSS Trading: www.fosstrading.com
Joshua Wiley
> I understand Josh's example:
>
> mat <- matrix(1:10, dimnames = list(NULL, "A"))
> cbind(X = 11:20, Y = mat + 1)
> cbind(X = 11:20, Y = mat[, "A"] + 1)
>
> In the line, cbind(X = 11:20, Y = mat + 1), it would be nice if an error or
> warning message were issued to the effect that the "Y = " part is ignored or
> not applicable or something to that effect.
> I still don't understand why cbind.xts doesn't do what I expect. If I try
> what Josh suggests on my xts example, the result is still confusing.
>
>> cbind.xts(close, A = close[,"close"])
>
> close close.1
>
> 2007-01-03 1416.60 1416.60
>
> 2007-01-04 1418.34 1418.34
>
> 2007-01-05 1409.71 1409.71
>
> 2007-01-08 1412.84 1412.84
>
> 2007-01-09 1412.11 1412.11
>
>
> If this helps:
>
>
> structure(c(1416.6, 1418.34, 1409.71, 1412.84, 1412.11), .indexCLASS =
> "Date", .indexTZ = structure("", .Names = "TZ"), src = "yahoo", updated =
> structure(1304875186.625, class = c("POSIXct",
>
>
> .Names = "TZ"), tclass = "Date"), .Dim = c(5L,
>
It does. Thank you. It sounds like you want to keep the xts class,
so this may not be useful to you, but if you are willing to drop it at
the point you are cbind()ing things together:
cbind(changed.close = as.matrix(close + 1)[, "close"], zero = 0, close)
should do the trick.
Cheers,
Josh
>
> I was also unable to convert Josh's matrix example to xts.
>
>> as.xts(mat, order.by = rownames())
>
> Error in dimnames(x) : 'x' is missing
>
> The order.by parameter is required, but I don't understand the error message
> and haven't been able to figure out how to fix it. If this helps:
>
>> dput(mat)
> structure(1:10, .Dim = c(10L, 1L), .Dimnames = list(NULL, "A"))
>
> This suggests that mat already has defined dimnames. So why does as.xts give
> an error message about dimnames? I was unable to figure out the problem by
> looking at any of the help files. In particular the help file
> for as.xts.methods didn't seem to offer any clarifications--at least none
> that I could understand.
>
> -- Russ
>
>
> On Sun, May 8, 2011 at 12:26 PM, Joshua Wiley <jwiley...@gmail.com>
> wrote:
>>
>> class(mat)
>> class(mat[, "A"])
>
--
Joshua Wiley
Ph.D. Student, Health Psychology
University of California, Los Angeles
http://www.joshuawiley.com/
______________________________________________
Jeff Ryan
We're of course getting into some incredibly fine-level detail on how all of
this works. I'll try and explain issues as I recall them over the
development of xts and cbind.xts
xts started as an extension of zoo. zoo is an extension of 'ts' (greatly
simplified comparison of course, but stay with me)
Achim and Gabor have put tremendous effort into the design of zoo - with a
primary focus on keeping it consistent with base R behavior. That is, try
not to introduce unnecessary changes to the interface an R user is
accustomed to. The logic being that this makes for a more consistent
interface as well as a easier learning curve and hence greater/faster
adoption rate.
'xts' extends this, though with a bit more flexibility in terms of
consistency. Why? Simply put - some things about R annoyed me coming from a
time-series background. Number one was the fact that lag() is backwards.
Backwards from expectation, nearly all literature, and all standard
definitions. So xts breaks with lag(, n=1) behavior. This is obviously
confusing to some - but was the gamble I was willing to take - consistency
(with R) be damned! ;-)
So, now back to cbind. cbind and merge in zoo-land (and xts by extension)
are the same. This isn't the case for other classes that use these - but
that is 'allowable' and 'expected' under a class dispatch system. The docs
for ?cbind state:
For ‘cbind’ (‘rbind’) the column (row) names are taken from the
‘colnames’ (‘rownames’) of the arguments if these are matrix-like.
Otherwise from the names of the arguments or where those are not
supplied and ‘deparse.level > 0’, by deparsing the expressions
given, for ‘deparse.level = 1’ only if that gives a sensible name
(a ‘symbol’, see ‘is.symbol’).
Based on that, I'd argue that xts does it "right". Of course I'll also point
out that this is incorrect thinking as well - since this is a description
for the generic - and not for xts. But again in a highly configurable
object/class system, where you start to make a distinction of right and
wrong is itself up for debate.
At the other end of the argument spectrum is _why not_. That is, why can't
cbind.xts handle the names to replace the colnames of objects passed in.
Here is where I'll point out that I am really just going by memory.
Three major items are involved in cbind. One is that dispatch is quite
unlike nearly every other dispatch in R. This is a fact - nothing to do
with xts.
* cbind isn't a generic (it's an .Internal call)
* it uses ...
* cbind can be called in numerous ways (I'll list only the common ones -
but with R you can do even crazier things)
do.call(cbind,
do.call(cbind.xts,
cbind,
cbind.xts,
merge,
merge.xts,
do.call(merge,
do.call(merge.xts
The rules of dispatch on cbind are really at a level that R-help has no
business discussing. The second part is where things actually get tricky
though. They all behave differently with respect to how args are handled -
when eval'd, etc.
I'm sure you have read how R strains itself on 'big data'. This is true and
false. Improper use (or just naive use) can cause object copies in places
you really don't want. Much of xts at this point is implemented in custom C
code. The gain here is that you can make it eas(ier) to avoid copies until
you need them by writing in C. Obvious, but needs to be said.
To figure out what the columns have - and if names are attached to the
objects in the pairlist (the "..." in this context) - you have to be very
careful. Touch anything in the wrong place or wrong time and you lose a
figurative arm and leg to memory copies. So, in 99.9999% of cases - where
you aren't naming (which would be an extra feature above and beyond c(olumn)
binding [the reason for cbind] - you run a very real risk of getting nailed
for copies you don't want. On 10MM obs that is almost manageable. On 100's
of millions or billions - it is kill -9 time.
To compound the issue - recall all of those different dispatch methods. Yep
- they all behave just a bit differently. How? Honestly - I don't know or
care. I simply know you can't easily make the behavior consistent amongst
those calls. I have tried. And tried.
End of day, and a very long R-help email, xts is different than base R. It
is even different than it's 'parent' zoo behavior. But in exchange for this
difference (and bit of learning/adjustment) you get a class that is faster
than anything else.
Period.
> x <- .xts(1:1e7, 1:1e7) # our time series object
> m <- coredata(x) # a matrix
> str(x)
An ‘xts’ object from 1969-12-31 18:00:01 to 1970-04-26 12:46:40 containing:
Data: int [1:10000000, 1] 1 2 3 4 5 6 7 8 9 10 ...
Indexed by objects of class: [POSIXt,POSIXct] TZ: America/Chicago
xts Attributes:
NULL
> str(m)
int [1:10000000, 1] 1 2 3 4 5 6 7 8 9 10 ...
> system.time(x[,1]) # get the first column
user system elapsed
0.017 0.000 0.017
> system.time(m[,1]) # ditto
user system elapsed
0.152 0.000 0.153
Yep, nearly 10x faster than a matrix op - AND you still have the time index.
To get there you need to sometimes make sacrifices. xts does, though I like
to think they are well thought out and consistent*
*enough ;-)
Best,
Jeff
--
Jeffrey Ryan
jeffre...@lemnica.com
[[alternative HTML version deleted]]
Russ Abbott
Clearly you (and others) have put a lot of work into xts -- and I'm the
beneficiary. So I'll stop complaining.
Thanks for the class (both code and explanation).
*-- Russ *