Excel Data Reader: Streaming

236 views
Skip to first unread message

Utopius

unread,
Feb 23, 2012, 11:27:22 PM2/23/12
to Linq To Excel
Using the LinqToExcel and Remotion DLLs, is it possible to query as a
stream, instead of an all-or-nothing buffer?

I'm trying to use LinqToExcel for extremely LARGE Excel datasets, and
I cannot load the columns into memory (in order to perform my
calculations) - without overloading the RAM available.

Is there a simple way to iterate over the Excel input in a streaming
manner, and form aggregate calculations (for various statistical
functions)...without overloading memory?

It was my understanding that LinqToExcel could (prepare to) read the
Excel data through the LINQ query (like a T-SQL cursor), pulling the
data only as it operated upon (via calculations, databinding, and/or
reporting operations). If I could read a single row of data, perform
some aggregate/cumulative operation, and move onto the next row while
disposing of the current row, I'd expect this to limit the in-process
memory footprint.

In other words, I thought the query won't execute until the data
source iterator performed 'actions' against the underlying query
results (much like a streaming SQLDataReader against a SQL DB).

Anyone able to set me straight?
How can I read very large Excel worksheets, in a streaming manner to
limit the memory consumption?

Utopius
(novice)

Paul Yoder

unread,
Feb 23, 2012, 11:33:58 PM2/23/12
to linqt...@googlegroups.com
It's not possible to stream in LinqToExcel since it executes the sql statement and loads all the results into memory.

However, you can stream the results by using an OLEDB connection to the spreadsheet.
Reply all
Reply to author
Forward
0 new messages