Stream was too long exception

[Mark Reid]

Running version 5.30, and trying to load a large (~300mb) excel stream via XlsxReader.
And getting thrown a Steam Was Too Long exception.

For reference, what is the stream size limitation in place here?

[shriop]

I've never seen this particular error message. It's not going to be an intended limitation.

Can you please provide the full stack trace and exact error message?

Xlsx is essentially a zip file, so your 300mb file could be as large as 3gb in memory once decompressed.

It sounds like this might be running into a max size limitation of MemoryStream, of around 2gb from some quick research, https://stackoverflow.com/questions/53193085/memory-limitted-to-about-2-5-gb-for-single-net-process .

If this is the case, I'd have to review to see if there's a way to stream data from the base stream instead of the copy, but it'd be helpful to know which method from the stack trace this is being thrown from.

Bruce Dunwiddie

[Mark Reid]

Stack trace here.
Definitely looks related to MemoryStream.

at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)  

    at DataStreams.Xlsx.InMemoryStreamCopy..ctor(Stream sourceStream)  

    at DataStreams.Xlsx.OOXml.Read.Workbook..ctor(Stream data)  

    at DataStreams.Xlsx.XlsxReader..ctor(Stream inputStream) 

[shriop]

Looking at why and how this code is being used internally, it's going to be fairly problematic to get it to work differently. I have some ideas, but I won't know until I try.

I don't know that I can even say at this point that I'll be able to get around this, let alone give you a proposed timeline.

If this is just a one time thing, can you just simply use Excel to save the file as a tab delimited file and process it instead? That process can handle a much larger file size.

Bruce Dunwiddie

[Mark Reid]

I noticed that upgrading from 5.24 to 5.30 has actually caused this issue.
Using the simplest of console apps, I can load the excel file in question under 5.24 but it dies with 5.30
FWIW, the uncompressed version of the xlsx is 3gig+

[shriop]

Well, that's pretty interesting.

There doesn't appear to have been a real reworking of the code during that time. I will have to research further to confirm.

The likely relevant change log items however are:

• Version 5.27 released.
  
  • Upgraded .Net Framework build to 4.5.2.
  • Added .Net Standard 2.0 build with compatibility for .Net Core 2.0+.

Are you running under .Net Framework or .Net Core, and if you're running under .Net Framework, are you using the .Net Framework build or the .Net Standard build?

Bruce Dunwiddie

[Mark Reid]

Running under framework 4.7.2

Using the .net framework build.

[shriop]

Please hit me up from your licensed email to my csvreader email address for an update to v5.31.

I have the library back able to handle bigger Xlsx files. There's still going to be a limit, but it's now back to only needing to store the original 300mb in a single stream in memory, as opposed to the full decompressed 3gb.

I looked back at each of the sets of changes between v5.24 and v5.30, and found that for v5.28 I swapped out some zip file functionality so that it would work under .Net Standard compatibility. In the process I changed some of how the streams were being worked with in memory to get it to work at the time. In reworking it now, I was able to be a little more creative and find a different path that works and retains the original capability.

Bruce Dunwiddie