Epic array and import to RnBeads

[Jenny]

Hello all,

I am analysing the new Illumina methylation array (Epic) and wonder if I can use the RnBeads package for this analysis.

I tried to import the data with:

data.source = c( idat.dir, sample.annotation)

report.dir = "reports/"

rnb.run.import(data.source=data.source, data.type="infinium.idat.dir", dir.reports=report.dir)

and it does not report any error but only the probes present on the 450K are import.

Is there a option where I can specify the array platform ?

Many thanks in advanced!

[Kasper Daniel Hansen]

I know this is not what is asked, but we have full (experimental) support for EPIC in minfi, see

  https://bitbucket.org/khansen/illumina_epic

Kasper

--
You received this message because you are subscribed to the Google Groups "Epigenomics forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to epigenomicsfor...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jenny]

Thanks Kasper :-)

[Jenny]

Do you have any clue when there is a new/updated minify version for the EPIC array?

[Kasper Daniel Hansen]

As I describe in the README on bitbucket, you can analyze EPIC arrays right now, with an additional line of code.

Changes to minfi will start by going into the devel branch; the next Bioconductor release is going to be in April.  I don't have a timeline for when I will make the changes in minfi devel (but it will be included in the next release); they are pretty minor but I have more pressing commitments, especially since the code in the README should make it possible for any user to do their analysis.

Best,

Kasper

[Fabian]

Hi Jenny,
we have implemented EPIC support and are currently testing it. So you can expect this feature in RnBeads very soon.

Best regards,
Fabian

[Jenny]

Sorry to bother you. I installed both the new annotation and manifest data and used the commands from the Readme file to adapted the read450k function. The EPIC data is perfectly loaded to R as a RGset. But all subsequent analysis can not be done. 

For example preprocessIllumina(RGset)

Error in getGreen(rgSet)[getProbeInfo(rgSet, type = "II")$AddressA, ] : subscript out of bounds 

or 

densityPlot(RGset)

Error in getGreen(rgSet)[getProbeInfo(rgSet, type = "II")$AddressA, ] : subscript out of bounds

Is this a major issue or can the code easily adopted. Many thanks for your kind help.

[Kasper Daniel Hansen]

Did you run the command changing the annotation slot in the rgset object after you loaded it?

[Jenny]

yes, I used:

RGset <- read.450k.exp(targets = targets)

RGset@annotation = c(array = "IlluminaHumanMethylationEPIC", annotation = "ilmn10.hg19")

RGset

RGChannelSet (storageMode: lockedEnvironment)

assayData: 1032279 features, 8 samples 

  element names: Green, Red 

An object of class 'AnnotatedDataFrame'

  sampleNames: 20.. (8 total)

  varLabels: array_ID Basename ... filenames (20 total)

  varMetadata: labelDescription

Annotation

  array: IlluminaHumanMethylationEPIC

  annotation: ilmn10.hg19

I also have the latest minifi version installed.

[Kasper Daniel Hansen]

To test your setup, download the Demo Data files from 

  http://support.illumina.com/array/array_kits/infinium-methylationepic-beadchip-kit/downloads.html

and see if you can read and preprocess them.

I have had one interaction with a group who was given an early file for the scanner, resulting in EPIC idat files which were "wrong".  It was corrected by re-scanning with an updated scanner file.  Of course, I may also have a bug.  Could you tell me the minfi version.

Best,

Kasper

[Jenny]

Dear Kasper, thanks a lot I tried the demo data and everything is fine. So I guess we scanned the data also wrongly. Thanks again for your helpful input!

[Kasper Daniel Hansen]

I'm curious about this since now I have heard about it two times.  If you can, I would love to get the IDAT files for one sample (red and green channel).

[Smeeta Shrestha]

Many Many Thanks !!! 

[Wei]

Dear Kasper,
I tried to export beta values using minfi and the Illumina EPIC array demo data set, but got some negative beta values.
Can you please help with pointing out what I did wrong? Thank you!

The following were my R codes and output:
require(minfi)
targets<-read.450k.sheet("/home/wei/vol/epic/analysis_ilmn_demo")
RGset<-read.450k.exp(targets=targets)
RGset@annotation <- c(array = "IlluminaHumanMethylationEPIC", annotation = "ilmn10.hg19")

MSet.norm<-preprocessIllumina(RGset,bg.correct=TRUE,normalize="controls",reference=1)
M <- getM(MSet.norm,type="beta",betaThreshold=0.001)
write.csv(M,file="ilmn_demo_beta.txt")

$ head -5 ilmn_demo_beta.txt
"","200144450019_R07C01","200144450021_R05C01"
"cg18478105",-5.73718997781117,-5.19785153256355
"cg09835024",-5.40622904166159,-4.91632985346493
"cg14361672",4.08136735039303,4.23840473932508
"cg01763666",2.57703725977641,2.17741633939611

Best,
-Wei

[Wei]

Just for your reference. I also emailed my question to Kasper. His email reply was the following which solved my problem.

"You're using getM which gives you logit(beta); these values can be negative. Just use getBeta.

Best,

Kasper

[Kasper Daniel Hansen]

And I replied in person before I saw he had posted to this forum.  In general, I don't like private emails with support questions.

Best,

Kasper

[Kasper Daniel Hansen]

The annotation and manifest packages for minfi have been updated, see

  https://bitbucket.org/khansen/illumina_epic

Since we have not detected in errors in the material from Illumina, I have submitted these packages to Bioconductor.  Note that the new annotation package has a slightly different name reflecting the new manifest package, ie.

  IlluminaHumanMethylationEPICanno.ilmn10b.hg19

instead of 

  IlluminaHumanMethylationEPICanno.ilmn10.hg19

When you read this files into R, you should now do 

  epicData@annotation <- c(array = "IlluminaHumanMethylationEPIC", annotation = "ilmn10b.hg19")

Best,

Kasper

[Kasper Daniel Hansen]

Link got changed to https://bitbucket.org/hansenlab/illumina_epic

[everso...@gmail.com]

Hi Everyone,

I'm also working with these EPIC array data. And I have run into an error when conducting functional normalization via preprocessFunnorm. I first encountered this error when processing some of our data, so I also tried downloading the EPIC demo data, and ran into the same error message when processing those. My code, and the error message are below.

> targets = read.450k.sheet("C://Users// ... //EPIC Data//infinium-methylationepic-demo-dataset//Demo Data EPIC")

> epicData= read.450k.exp(targets = targets)

> epicData@annotation = c(array = "IlluminaHumanMethylationEPIC", annotation = "ilmn10b.hg19")

> Mset.norm = preprocessFunnorm(epicData, sex = NULL, bgCorr = T, dyeCorr = T, nPCs = 2, verbose = TRUE)

[preprocessFunnorm] Background and dye bias correction with noob 

[preprocessNoob] Using sample number 2 as reference level...

[preprocessFunnorm] Mapping to genome

[preprocessFunnorm] Quantile extraction

[preprocessFunnorm] Normalization

Error in greenControls[[index]][addr, ] : subscript out of bounds

I am using R version 3.2.3, updated the minfi package and all required packages, and am using the most recent EPIC annotation and manifest zip files provided by Kasper via bitbucket. Has anyone else run into this issue, or have any suggestions?

Thanks!

-Todd

[Kasper Daniel Hansen]

Thanks for the report.  This has been fixed in minfi 1.17.6 which I have just uploaded to Bioconductor; should be available Wednesday morning US eastern time.  This will not be fixed in the release version of minfi which you are using; you will need to switch to minfi devel (which requires you to use R-devel).

Best,

Kasper

[Kasper Daniel Hansen]

Btw. its caused by the v1.0b manifest removing a bisulfite conversion I control probe, presumably because it was not working very well.  Why Illumina only decided to remove this probe from the EPIC manifest and not the 450k manifest I don't know; presumably its the same probe on the two arrays although I don't know for sure.

Best,

Kasper

[everso...@gmail.com]

Great, thanks for the update!

[vk]

Hi Jenny,

Which package did you use for Illumina methylation EPIC array analysis? I have samples from this array. I have IDAT files. I need to do differential methylation analysis and also need to get QC plots. Could you please tell me how to do the analysis and which package is best for that. Thank you

[Adrian B]

When preprocessing IDAT-files from the IlluminaHumanMethylationEPIC array, I successfully load the RGChannelSet but get an error when attempting to run the following functions 'preprocessRaw', 'qcReport', 'getBeta' etc. The error says: "Error in getGreen(rgSet)[getProbeInfo(rgSet, type = "II")$AddressA, ] : subscript out of bounds". When I run the same script on the DemoData files, everything works perfectly. In the Google Groups forum, you write: "I have had one interaction with a group who was given an early file for the scanner, resulting in EPIC idat files which were "wrong".  It was corrected by re-scanning with an updated scanner file." - given that we run into the same problem, do you think this error can be adjusted by re-scanning the IDAT-files?

minfi version 1.18.2

R code: 

setwd("C/XYZ")

library(minfi)

library(IlluminaHumanMethylationEPICanno.ilm10b2.hg19)

library(IlluminaHumanMehylationEPICmanifest)

memory.limit(size=40000)

targets <- read.table("targets.txt",sep="\t",header=TRUE)

targets$Basename <- as.character(targets$Basename)

idatPath <- setwd("C:/XYZ/IDAT files")

RGSet <- read.metharray.exp(base=idatPath,targets=targets)

#Here, annotation(RGSet) returns "unknown array" & "annotation xx" in our original data, but with the DemoData files it correctly specifies "array #IlluminaHumanMethylationEPIC" & "annotation ilm10b2.hg19"

RGSet@annotation <- c(array="IlluminaHumanMethylationEPIC",annotation="ilm102.hg19")

MSet.norm<-preprocessIllumina(RGSet,bg.correct=TRUE,normalize="controls",reference=1)

#"Error in getGreen(rgSet)[getProbeInfo(rgSet, type = "II")$AddressA, ] : subscript out of bounds"

[Evadnie Rampersaud]

Hi,

I just ran into a similar error. Hoping someone replies. 

> gset.quantile <- preprocessQuantile(RGSet, fixOutliers = TRUE, removeBadSamples = TRUE, badSampleCutoff = 10.5, quantileNormalize = TRUE, stratified = TRUE, mergeManifest = FALSE, sex = NULL)

[preprocessQuantile] Mapping to genome.

[preprocessQuantile] Fixing outliers.

[preprocessQuantile] Quantile normalizing.

>

> gset.funnorm <- preprocessFunnorm(RGSet)

[preprocessFunnorm] Background and dye bias correction with noob

[preprocessNoob] Using sample number 6 as reference level...

[preprocessFunnorm] Mapping to genome

[preprocessFunnorm] Quantile extraction

[preprocessFunnorm] Normalization

Error in greenControls[[index]][addr, ] : subscript out of bounds

[Jean-Philippe Fortin]

Which version of minfi are you using?  

[erampers]

Hi thanks,

minfi version: 1.16.1

[erampers]

Hi I am using  minfi version: 1.16.1

On Wednesday, July 27, 2016 at 7:27:05 AM UTC-5, Jean-Philippe Fortin wrote:

[Jean-Philippe Fortin]

You should not encounter this problem for minfi version > 1.18.2

[erampers]

Thanks, i realized I was using an older version compared to the person who also wrote before me. I'll try the current version, thanks!

[erampers]

Hi there,

The wrong version of Minfi seems to be downloading via BioCLite. Any recommendations? thanks 

[erampers]

Looks like an older version is downloading

> source("https://bioconductor.org/biocLite.R")

Bioconductor version 3.2 (BiocInstaller 1.20.3), ?biocLite for help

A new version of Bioconductor is available after installing the most recent version of R; see http://bioconductor.org/install

> biocLite("minfi")

BioC_mirror: https://bioconductor.org

Using Bioconductor 3.2 (BiocInstaller 1.20.3), R 3.2.4 Revised (2016-03-16 r70336).

Installing package(s) ‘minfi’

also installing the dependency ‘matrixStats’

trying URL 'https://cran.rstudio.com/bin/windows/contrib/3.2/matrixStats_0.50.2.zip'

Content type 'application/zip' length 1529865 bytes (1.5 MB)

downloaded 1.5 MB

trying URL 'https://bioconductor.org/packages/3.2/bioc/bin/windows/contrib/3.2/minfi_1.16.1.zip'

Content type 'application/zip' length 1363505 bytes (1.3 MB)

downloaded 1.3 MB

package ‘matrixStats’ successfully unpacked and MD5 sums checked

package ‘minfi’ successfully unpacked and MD5 sums checked

The downloaded binary packages are in

        C:\Users\erampers\AppData\Local\Temp\RtmpYjdDlY\downloaded_packages

Old packages: 'cluster', 'Matrix', 'mgcv', 'nlme', 'survival'

Update all/some/none? [a/s/n]: n

> rm(list=ls(all=TRUE))

> options(stringsAsFactors = FALSE);

> library(minfi)

Loading required package: BiocGenerics

Loading required package: parallel

Attaching package: ‘BiocGenerics’

The following objects are masked from ‘package:parallel’:

    clusterApply, clusterApplyLB, clusterCall, clusterEvalQ, clusterExport, clusterMap, parApply, parCapply, parLapply, parLapplyLB, parRapply, parSapply, parSapplyLB

The following objects are masked from ‘package:stats’:

    IQR, mad, xtabs

The following objects are masked from ‘package:base’:

    anyDuplicated, append, as.data.frame, as.vector, cbind, colnames, do.call, duplicated, eval, evalq, Filter, Find, get, grep, grepl, intersect, is.unsorted, lapply, lengths, Map, mapply, match, mget, order,

    paste, pmax, pmax.int, pmin, pmin.int, Position, rank, rbind, Reduce, rownames, sapply, setdiff, sort, table, tapply, union, unique, unlist, unsplit

Loading required package: Biobase

Welcome to Bioconductor

    Vignettes contain introductory material; view with 'browseVignettes()'. To cite Bioconductor, see 'citation("Biobase")', and for packages 'citation("pkgname")'.

Loading required package: lattice

Loading required package: GenomicRanges

Loading required package: S4Vectors

Loading required package: stats4

Loading required package: IRanges

Loading required package: GenomeInfoDb

Loading required package: SummarizedExperiment

Loading required package: Biostrings

Loading required package: XVector

Loading required package: bumphunter

Loading required package: foreach

foreach: simple, scalable parallel programming from Revolution Analytics

Use Revolution R for scalability, fault tolerance and more.

http://www.revolutionanalytics.com

Loading required package: iterators

Loading required package: locfit

locfit 1.5-9.1   2013-03-22

Setting options('download.file.method.GEOquery'='auto')

Setting options('GEOquery.inmemory.gpl'=FALSE)

Warning messages:

1: package ‘foreach’ was built under R version 3.2.5 

2: package ‘locfit’ was built under R version 3.2.5 

> 

> packageVersion("minfi")

[1] ‘1.16.1’

[Kasper Daniel Hansen]

Update R

Best,

Kasper

(Sent from my phone.)

[aniruddha chatterjee]

Dear Rnbead team,

I am trying to work though the package. As a start I have tried the Ziller dataset and with this command it worked fine.

## Trying with Ziller dataset #

data.dir <- "/Volumes/dsm-pathology-genomics-1/Aniruddha_Chatterjee/Working/Ziller2011_PLoSGen_450K"
idat.dir <- file.path(data.dir, "idat")
sample.annotation <- file.path(data.dir, "sample_annotation.csv")
# Directory where the output should be written to
analysis.dir <- "/Volumes/dsm-pathology-genomics-1/Aniruddha_Chatterjee/Working/Ziller2011_PLoSGen_450K/analysis"# Directory where the report files should be written to report.dir <- file.path(analysis.dir, "reports")

# Directory where the report files should be written to #
report.dir <- file.path(analysis.dir, "reports")

# Re-intitilaze a report directory for tailor analysis ##
report.dir <- file.path(analysis.dir, "reports_details")
rnb.initialize.reports(report.dir)
#[1] FALSE
logger.start(fname=NA)

data.source <- c(idat.dir, sample.annotation)
source("https://bioconductor.org/biocLite.R")
biocLite("RnBeads.hg19")
result <- rnb.run.import(data.source=data.source,data.type="infinium.idat.dir", dir.reports=report.dir)
#worked fine#
rnb.set <- result$rnb.set
rnb.set ## see how it looks.
#Object of class RnBeadRawSet
12 samples
485577 probes
of which: 482421 CpG, 3091 CpH, and 65 rs
Region types:
  138358 regions of type tiling
31033 regions of type genes
31195 regions of type promoters
26662 regions of type cpgislands
Intensity information is present
Detection p-values are present
Bead counts are present
Quality control information is present
Summary of normalization procedures:
  The methylation data was not normalized.
No background correction was performed.##

# Visualize the bimodal distribution of beta values in sample 5#
hist(meth(rnb.set)[, 5])
# WORKS FINE

Then I tried importing my one dataset in similar fashion. I have 21 sample and I have checked that I have 42 IDAT files (21 grn and 21 red), its exactly same as shown in Zillers. However, I am getting this error.


## set directory where 450K data is there ##
library(RnBeads)
data.dir <- "/Volumes/dsm-pathology-genomics-1/Aniruddha_Chatterjee/Working/450K_Shohei"
idat.dir <- file.path(data.dir, "21_idat")
sample.annotation <- file.path(data.dir, "sample_annotation_HB_mod.csv")

# Directory where the report files should be written to report.dir <- file.path(analysis.dir, "reports")#
analysis.dir <- "/Volumes/dsm-pathology-genomics-1/Aniruddha_Chatterjee/Working/450K_Shohei/analysis"

# Directory where the report files should be written to #
report.dir <- file.path(analysis.dir, "reports")

# Re-intitilaze a report directory for tailor analysis ##
report.dir <- file.path(analysis.dir, "reports_details")
rnb.initialize.reports(report.dir)

# logger.start(fname=NA) # if you dont want report #
data.source <- c(idat.dir, sample.annotation)
result <- rnb.run.import(data.source=data.source,data.type="infinium.idat.dir", dir.reports=report.dir)

## error message:

2017-02-08 13:34:06     0.4  STATUS                 STARTED Loading Data
2017-02-08 13:34:07     0.4    INFO                     Number of cores: 1
2017-02-08 13:34:09     0.4    INFO                     Loading data of type "infinium.idat.dir"
2017-02-08 13:34:10     0.4  STATUS                     STARTED Loading Data from IDAT Files
2017-02-08 13:34:12     0.4 WARNING                         The sample sheet table has only one column. Please check import.table.separator option
2017-02-08 13:34:13     0.4    INFO                         Added column barcode to the provided sample annotation table
2017-02-08 13:34:14     0.4   ERROR                         Some IDAT files are not present in the supplied base directory, for instance 1,2.00E+11,R01C01,2,N,2N_1,2.00E+11,R01C01,2,N,2N_Red.idat, 1,2.00E+11,R01C01,2,N,2N_1,2.00E+11,R01C01,2,N,2N_Grn.idat, 2,2.00E+11,R02C01,2,E,2E_2,2.00E+11,R02C01,2,E,2E_Red.idat, 2,2.00E+11,R02C01,2,E,2E_2,2.00E+11,R02C01,2,E,2E_Grn.idat, 3,2.00E+11,R03C01,2,M,2M_3,2.00E+11,R03C01,2,M,2M_Red.idat, 3,2.00E+11,R03C01,2,M,2M_3,2.00E+11,R03C01,2,M,2M_Grn.idat
Error in logger.error(txt) :
Some IDAT files are not present in the supplied base directory, for instance 1,2.00E+11,R01C01,2,N,2N_1,2.00E+11,R01C01,2,N,2N_Red.idat, 1,2.00E+11,R01C01,2,N,2N_1,2.00E+11,R01C01,2,N,2N_Grn.idat, 2,2.00E+11,R02C01,2,E,2E_2,2.00E+11,R02C01,2,E,2E_Red.idat, 2,2.00E+11,R02C01,2,E,2E_2,2.00E+11,R02C01,2,E,



> sessionInfo()
R version 3.3.2 (2016-10-31)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: OS X Yosemite 10.10.5

locale:
[1] en_NZ.UTF-8/en_NZ.UTF-8/en_NZ.UTF-8/C/en_NZ.UTF-8/en_NZ.UTF-8

attached base packages:
 [1] grid      stats4    parallel  stats     graphics
 [6] grDevices utils     datasets  methods   base    

other attached packages:
 [1] RnBeads.hg19_1.6.0                    
 [2] BiocInstaller_1.24.0                  
 [3] RnBeads_1.6.1                         
 [4] plyr_1.8.4                            
 [5] methylumi_2.20.0                      
 [6] minfi_1.20.2                          
 [7] bumphunter_1.14.0                     
 [8] locfit_1.5-9.1                        
 [9] iterators_1.0.8                       
[10] foreach_1.4.3                         
[11] Biostrings_2.42.1                     
[12] XVector_0.14.0                        
[13] SummarizedExperiment_1.4.0            
[14] FDb.InfiniumMethylation.hg19_2.2.0    
[15] org.Hs.eg.db_3.4.0                    
[16] TxDb.Hsapiens.UCSC.hg19.knownGene_3.2.2
[17] GenomicFeatures_1.26.2                
[18] AnnotationDbi_1.36.2                  
[19] reshape2_1.4.2                        
[20] scales_0.4.1                          
[21] Biobase_2.34.0                        
[22] illuminaio_0.16.0                     
[23] matrixStats_0.51.0                    
[24] limma_3.30.10                         
[25] gridExtra_2.2.1                       
[26] gplots_3.0.1                          
[27] ggplot2_2.2.1                         
[28] fields_8.10                           
[29] maps_3.1.1                            
[30] spam_1.4-0                            
[31] ff_2.2-13                             
[32] bit_1.1-12                            
[33] cluster_2.0.5                         
[34] RColorBrewer_1.1-2                    
[35] MASS_7.3-45                           
[36] GenomicRanges_1.26.2                  
[37] GenomeInfoDb_1.10.3                   
[38] IRanges_2.8.1                         
[39] S4Vectors_0.12.1                      
[40] BiocGenerics_0.20.0                   

loaded via a namespace (and not attached):
 [1] nlme_3.1-131             bitops_1.0-6           
 [3] httr_1.2.1               tools_3.3.2            
 [5] doRNG_1.6                nor1mix_1.2-2          
 [7] R6_2.2.0                 KernSmooth_2.23-15     
 [9] DBI_0.5-1                lazyeval_0.2.0         
[11] colorspace_1.3-2         base64_2.0             
[13] preprocessCore_1.36.0    pkgmaker_0.22          
[15] rtracklayer_1.34.1       caTools_1.17.1         
[17] genefilter_1.56.0        quadprog_1.5-5         
[19] stringr_1.1.0            digest_0.6.12          
[21] Rsamtools_1.26.1         siggenes_1.48.0        
[23] GEOquery_2.40.0          RSQLite_1.1-2          
[25] mclust_5.2.2             BiocParallel_1.8.1     
[27] gtools_3.5.0             RCurl_1.95-4.8         
[29] magrittr_1.5             Matrix_1.2-8           
[31] Rcpp_0.12.9              munsell_0.4.3          
[33] stringi_1.1.2            zlibbioc_1.20.0        
[35] gdata_2.17.0             lattice_0.20-34        
[37] splines_3.3.2            multtest_2.30.0        
[39] annotate_1.52.1          beanplot_1.2           
[41] rngtools_1.2.4           codetools_0.2-15       
[43] biomaRt_2.30.0           XML_3.98-1.5           
[45] data.table_1.10.4        gtable_0.2.0           
[47] openssl_0.9.6            reshape_0.8.6          
[49] assertthat_0.1           xtable_1.8-2           
[51] survival_2.40-1          tibble_1.2             
[53] GenomicAlignments_1.10.0 registry_0.3           
[55] memoise_1.0.0


I will really appreciate your help. Thank you in advance.