A few million values on x-axis
thomas
I am trying to plot a chart that has 3.8 million entries on the x-
axis. The y-axis only has 20 distinct ones.
I tried to simply plot the matrix, which I read through csv. Two
columns, first is a counter column (1:3800000), second has the y-axis
values.
Plotting it in matlab takes less than a second. With ggplot2 it
doesn't seem to finish at all.
What am I doing wrong here?
data <- read.csv("cat.txt")
p <- ggplot(data,aes(x=x,y=y)) + geom_line(colour=lcolor)
print(p)
Data looks like this:
x,y
1,1
2,1
3,1
4,2
5,4
....
Thank in advance ;)
Thomas
Claudia Beleites
Not knowing anything about your data I can produce a black rectangle in 11 s
from a 1/40 of your data size.
> x <- 1 : 1e5
> y <- sample (1:20, length (x), replace = TRUE)
> system.time (plot (x, y, type = "l"))
user system elapsed
4.080 0.020 4.097
> p <- ggplot(data.frame (x = x, y = y),aes(x=x,y=y)) + geom_line()
> system.time (print (p))
user system elapsed
10.59 0.03 10.65
However, plotting 4e6 values along x means some thousand values compressed into
one screen pixel. So: if you don't have a kind of structure on that scale,
you're not going to see anything with the line approach.
=> in that case: you need to compute some summary statistic (tiles, running
average, ...). Without knowing what your data means, no one can tell what does
make sense and what doesn't.
If the data is (somewhat) sorted, you can use rle()
y <- sort (y)
> system.time ({
+ yr <- rle (y)
+ data <- data.frame (x = c (0, cumsum (yr$lengths)),
+ y = c (1, yr$values))
+ p <- ggplot(data,aes(x=x,y=y)) + geom_step(direction = "vh")
+ print (p)
+ })
user system elapsed
0.960 0.000 0.973
HTH,
Claudia
--
Claudia Beleites
Dipartimento dei Materiali e delle Risorse Naturali
Universit� degli Studi di Trieste
Via Alfonso Valerio 6/a
I-34127 Trieste
phone: +39 0 40 5 58-37 68
email: cbel...@units.it
Brian Diggs
you expect to get from this plot. So what you may be doing wrong is
trying to plot 3.8 million points/line segments.
> > data <- read.csv("cat.txt")
> > p <- ggplot(data,aes(x=x,y=y)) + geom_line(colour=lcolor)
> > print(p)
>
> > Data looks like this:
> > x,y
> > 1,1
> > 2,1
> > 3,1
> > 4,2
> > 5,4
> > ....
to geom_point(). Instead of a rectangle, I got 20 effectively solid
lines across the plot.
tim <- function(n) {
data <- data.frame(x=1:n, y=sample(1:20, n, replace=TRUE))
print(system.time(p <- ggplot(data, aes(x=x,y=y)) + geom_point()))
print(system.time(print(p)))
}
tim(3800)
# user system elapsed
# 0.01 0.00 0.02
# user system elapsed
# 1.83 0.08 1.91
tim(38000)
# user system elapsed
# 0.00 0.02 0.02
# user system elapsed
# 4.02 0.48 4.50
tim(380000)
# user system elapsed
# 0.04 0.00 0.04
# user system elapsed
# 30.58 4.39 34.97
tim(3800000)
# user system elapsed
# 0.25 0.14 0.39
#Error: cannot allocate vector of size 29.0 Mb
#In addition: There were 50 or more warnings (use warnings() to see
the first 50)
#Timing stopped at: 347.13 15.63 362.91
The vast majority of the time is spent putting the plot on the screen,
not in creating the plot structure. Also, my machine was not even
able to make a plot with 3.8 million points.
As Claudia said, think about what you want the plot to show, and then
summarize the data to show that particular aspect.
> > Thank in advance ;)
>
> > Thomas
>
> --
> Claudia Beleites
> Dipartimento dei Materiali e delle Risorse Naturali
> Universit degli Studi di Trieste
> Via Alfonso Valerio 6/a
> I-34127 Trieste
>
> phone: +39 0 40 5 58-37 68
--
Brian Diggs
Senior Research Associate, Department of Surgery
Oregon Health & Science University
Mark Connolly
> As Claudia said, we don't have your data and so it is hard to say what
> you expect to get from this plot. So what you may be doing wrong is
> trying to plot 3.8 million points/line segments.
>
baptiste auguie
segments are much faster than ggplot2 at drawing. Below is a
comparison with and without grouping. It seems the slowdown is due to
the calculations prior to drawing.
baptiste
N <- 1e5
d <- data.frame(x=seq(1,N), y=rnorm(N), id=rep(seq(1,N/2),each=2))
## version for segments
xx <- rep(d$x, each=2)[-c(1, 2*N)]
yy <- rep(d$y, each=2)[-c(1, 2*N)]
start <- seq(1, by=2, length(xx)) # starting points
end <- seq(2, by=2, length(yy)) # end points
d2 <- data.frame(x0=xx[start], x1=xx[end],
y0=yy[start], y1=yy[end],
id=rep(seq(1,N-1),each=2))
## drawing
p1 <- qplot(x,y,data=d,geom="line")
p2 <- p1 + aes(group=id)
g1 <- with(d, polylineGrob(x/N, y/2))
g2 <- with(d, polylineGrob(x/N, y/2, id=id))
g3 <- with(d2, segmentsGrob(x0/N,y0/2,x1/N,y1/2, gp=gpar(col=id)))
png()
system.time(print(p1))
system.time(print(p2))
grid.newpage()
system.time(grid.draw(g1))
grid.newpage()
system.time(grid.draw(g2))
grid.newpage()
system.time(grid.draw(g3))
dev.off()
## user system elapsed
## 9.657 1.118 11.022
## > user system elapsed
## 117.253 23.795 144.114
## > > user system elapsed
## 0.729 0.009 0.753
## > > user system elapsed
## 0.849 0.019 0.885
## > > user system elapsed
## 3.094 0.018 3.134
> --
> You received this message because you are subscribed to the ggplot2 mailing
> list.
> Please provide a reproducible example: http://gist.github.com/270442
>
> To post: email ggp...@googlegroups.com
> To unsubscribe: email ggplot2+u...@googlegroups.com
> More options: http://groups.google.com/group/ggplot2
>
thomas
Mark, no, unfortunately i really meant it ;)
Best,
Thomas
Mark Connolly
plotter? I wonder if matlab determines that the result will be much
denser than the display can resolve, short circuits the operation, and
renders a filled polygon of some sort?
thomas
using Claudia's approach.
It is just a line, why would it plot a filled polygon?
Claudia Beleites
you forgot to tell the others that the y values are actually ordered.
Claudia
thomas
true, and next time I will put a picture of the plot online, will be
more accurate than my explanation attempts ;)
Thomas