creating simple plot ends up in a memory exhausted error

[Herbert Jägle]

I have a data.frame with 6 columns and 4630 rows. The data has been
created from multiple files and saved.
Using following code to read and plot the data results in a "memory
exhausted" error.

Here is the plotting code:
--------------
load('resc')
print(summary(resc))

print('..fl30-snr30-vs-snr60.pdf..')
xlstr <- expression('30 Hz SNR')
ylstr <- expression('60 Hz SNR')
p <- ggplot(resc) +
geom_point(aes(snr.30, snr.60, colour=fact.30)) +
scale_x_continuous(name=xlstr) +
scale_y_continuous(name=ylstr) +
scale_colour_manual('', values=c('grey', 'black'), legend=FALSE) +
facet_grid(. ~ stype)
theme_set(theme_bw())
p
ggsave(file='fl30-snr30-vs-snr60.pdf',width=8, height=3)
--------------



A summary of the data (i will email the original data file on request)
and the error message:
--------------
fn stype eye segment snr.
30
N1 : 122 N:1708 OS:2318 Min. : 1 0.0185913534345293:
1
N2 : 122 P:2928 OD:2318 1st Qu.:16 0.0203365470772822:
1
N3 : 122 Median :31 0.021445811881021 :
1
N4 : 122 Mean :31 0.0496526152000194:
1
N5 : 122 3rd Qu.:46 0.0545743752348621:
1
N6 : 122 Max. :61 0.0580861714029565:
1
(Other):3904 (Other) :
4630
fact.30 snr.60 fact.60
not sign.:3104 0.005353479967123 : 1 not sign.:4272
sign. :1532 0.00767336215167676: 1 sign. : 364
0.00797185376465362: 1
0.00879009426650364: 1
0.0190637314906903 : 1
0.0243221991847799 : 1
(Other) :4630
[1] "..fl30-snr30-vs-snr60.pdf.."
Error: memory exhausted (limit reached?)
In addition: Warning messages:
1: In paste(rep(l, length(lvs)), rep(lvs, each = length(l)), sep =
sep) :
Reached total allocation of 1535Mb: see help(memory.size)
Calls: source ... stamp -> cast -> eval -> add_group -> interaction ->
paste
2: In paste(rep(l, length(lvs)), rep(lvs, each = length(l)), sep =
sep) :
Reached total allocation of 1535Mb: see help(memory.size)
Calls: source ... stamp -> cast -> eval -> add_group -> interaction ->
paste
Error: cannot allocate vector of size 2 Kb
In addition: Warning messages:
1: Reached total allocation of 1535Mb: see help(memory.size)
2: Reached total allocation of 1535Mb: see help(memory.size)
--------------



I also tried the simple plot which also ends with the memory error:
--------------
load('resc')
print('..fl30-snr30-vs-snr60.pdf..')
xlstr <- expression('30 Hz SNR')
ylstr <- expression('60 Hz SNR')
p <- ggplot(resc) +
geom_point(aes(snr.30, snr.60)) +
scale_x_continuous(name=xlstr) +
scale_y_continuous(name=ylstr)
theme_set(theme_bw())
p
ggsave(file='fl30-snr30-vs-snr60.pdf',width=8, height=3)
--------------

Any suggestions what could be the problem?
Herbert

[Dennis Murphy]

Hi:

Looks like eight columns to me, but I digress :)

A file like this shouldn't take up more than 640K, so even with only 1.5Gb to play with, it shouldn't be impossible to get this to work. What other objects do you have in your workspace and how much room do they take up?

Dennis

--
You received this message because you are subscribed to the ggplot2 mailing list.
Please provide a reproducible example: http://gist.github.com/270442

To post: email ggp...@googlegroups.com
To unsubscribe: email ggplot2+u...@googlegroups.com
More options: http://groups.google.com/group/ggplot2

[Herbert Jägle]

Hi, i am loading just the required libraries and some functions i use.

after startup and loading default libraries:
memory.size()
[1] 14.33
> memory.limit()
[1] 510
load('resc')
> memory.size()
[1] 15.4

so almost no memory used.
Herbert

On 3 Nov., 23:57, Dennis Murphy <djmu...@gmail.com> wrote:
> Hi:
>
> Looks like eight columns to me, but I digress :)
>
> A file like this shouldn't take up more than 640K, so even with only 1.5Gb
> to play with, it shouldn't be impossible to get this to work. What other
> objects do you have in your workspace and how much room do they take up?
>
> Dennis
>

> > To unsubscribe: email ggplot2+u...@googlegroups.com<ggplot2%2Bunsu...@googlegroups.com>
> > More options:http://groups.google.com/group/ggplot2

[Dennis Murphy]

Hi:

After getting the data (thank you), I found that it seemed to have a rather odd characteristic:

str(resc)
List of 8
 $ fn     : Factor w/ 38 levels "N1","N2","N3",..: 1 1 1 1 1 1 1 1 1 1 ...
 $ stype  : Factor w/ 2 levels "N","P": 1 1 1 1 1 1 1 1 1 1 ...
 $ eye    : Factor w/ 2 levels "OS","OD": 1 1 1 1 1 1 1 1 1 1 ...
 $ segment: num [1:4636] 1 2 3 4 5 6 7 8 9 10 ...
 $ snr.30 : Factor w/ 4636 levels "0.0185913534345293",..: 4018 3199 2699 4103 2796 2603 2880 3048 3393 3138 ...
  ..- attr(*, "names")= chr [1:4636] "N1_N_OS_1" "N1_N_OS_2" "N1_N_OS_3" "N1_N_OS_4" ...
 $ fact.30: Factor w/ 2 levels "not sign.","sign.": 2 2 1 2 1 2 1 1 2 1 ...
  ..- attr(*, "names")= chr [1:4636] "N1_N_OS_1" "N1_N_OS_2" "N1_N_OS_3" "N1_N_OS_4" ...
 $ snr.60 : Factor w/ 4636 levels "0.005353479967123",..: 1605 592 3163 3016 533 3053 1024 2176 611 3399 ...
  ..- attr(*, "names")= chr [1:4636] "N1_N_OS_1" "N1_N_OS_2" "N1_N_OS_3" "N1_N_OS_4" ...
 $ fact.60: Factor w/ 2 levels "not sign.","sign.": 1 1 1 1 1 1 1 1 1 1 ...
  ..- attr(*, "names")= chr [1:4636] "N1_N_OS_1" "N1_N_OS_2" "N1_N_OS_3" "N1_N_OS_4" ...
 - attr(*, "row.names")= int [1:4636] 1 2 3 4 5 6 7 8 9 10 ...
 - attr(*, "idvars")= chr [1:4] "fn" "stype" "eye" "segment"
 - attr(*, "rdimnames")=List of 2                                                    <<<<<<<<<<=======================
  ..$ :'data.frame':    4636 obs. of  4 variables:
  .. ..$ fn     : Factor w/ 38 levels "N1","N2","N3",..: 1 1 1 1 1 1 1 1 1 1 ...
  .. ..$ stype  : Factor w/ 2 levels "N","P": 1 1 1 1 1 1 1 1 1 1 ...
  .. ..$ eye    : Factor w/ 2 levels "OS","OD": 1 1 1 1 1 1 1 1 1 1 ...
  .. ..$ segment: num [1:4636] 1 2 3 4 5 6 7 8 9 10 ...
  ..$ :'data.frame':    4 obs. of  2 variables:
  .. ..$ freq    : num [1:4] 30 30 60 60
  .. ..$ variable: Factor w/ 2 levels "snr","fact": 1 2 1 2

Your resc object seems to have two data frames nested within the rdimnames attribute, which didn't look like a good sign to me. I'm pretty sure that's the source of your inability to get the plot. That the two variables you wanted to plot were both factors didn't help either. I decided to simplify matters by copying the first eight components to a data frame resc2 and cleaning it up a bit:

resc2 <- resc[, 1:8]
str(resc2)   # before
resc2$snr.30 <- as.numeric(as.character(unname(resc2$snr.30)))
resc2$snr.60 <- as.numeric(as.character(unname(resc2$snr.60)))
resc2$fact.30 <- unname(resc2$fact.30)
resc2$fact.60 <- unname(resc2$fact.60)
str(resc2)  # after

This takes the 4636 level 'factors' snr.30 and snr.60 and converts them to numeric. I also unnamed the four variables above to simplify matters. Once this was done, the plot was straightforward:

xlstr <- '30 Hz SNR'
ylstr <- '60 Hz SNR'
p <- ggplot(resc2) + theme_bw() +
     geom_point(aes(x = snr.30, y = snr.60, colour=fact.30)) +

     scale_x_continuous(name=xlstr) +
     scale_y_continuous(name=ylstr) +
     scale_colour_manual('', values=c('grey', 'black'), legend=FALSE) +
     facet_grid(. ~ stype)

p
system.time(print(p))
   user  system elapsed
   1.17    0.04    1.23

HTH,
Dennis

More options: http://groups.google.com/group/ggplot2

[Herbert Jägle]

Dennis, thanks for looking into the data and the solution!!

I am wondering about how to avoid this problem. The data was read from
files and then reshaped using following code:

-code 1---------
res$fact.30 <- factor((res$p_limit.30 <= 0.05), levels=c(FALSE, TRUE),
labels=c('not sign.', 'sign.'))
resm <- melt(res, id.vars=c('fn', 'stype', 'eye', 'segment', 'freq'),
measure.vars=c('snr', 'fact'))
resc <- cast(resm, fn + stype + eye + segment ~ freq + variable)
names(resc) <- c('fn', 'stype', 'eye', 'segment', 'snr.30', 'fact.30',
'snr.60', 'fact.60')
----------------

If i omit the factor res$fact in the melt and cast statements and
recreate resc$fact.30 from p_limit (see code2), then everything works
fine. No prolem with plotting.

-code 2---------
resm <- melt(res, id.vars=c('fn', 'stype', 'eye', 'segment', 'freq'),
measure.vars=c('snr', 'p_limit'))
resc <- cast(resm, fn + stype + eye + segment ~ freq + variable)
names(resc) <- c('fn', 'stype', 'eye', 'segment', 'snr.30', 'p_limit.
30',
'snr.60', 'p_limit.60')
resc$fact.30 <- factor((resc$p_limit.30 <= 0.05), levels=c(FALSE,
TRUE),
labels=c('not sign.', 'sign.'))
----------------

Actually, i do not understand why code2 works fine and code1 ends in a
memory error. Could it be a cast related problem?

Best,
Herbert

[Hadley Wickham]

Hi Herbert,

Could you try with reshape2 and dcast and see if you still get the same problem?

Hadley

> --
> You received this message because you are subscribed to the ggplot2 mailing list.
> Please provide a reproducible example: http://gist.github.com/270442
>
> To post: email ggp...@googlegroups.com
> To unsubscribe: email ggplot2+u...@googlegroups.com
> More options: http://groups.google.com/group/ggplot2
>

--
Assistant Professor / Dobelman Family Junior Chair
Department of Statistics / Rice University
http://had.co.nz/