Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
import issue
8 views
Skip to first unread message
Francisco Gutierrez
Dec 20, 2011, 3:03:00 AM12/20/11
to
Dear Group:
I am having trouble importing large files from Excel (or csv) into Mathematica. The program gets stuck or crashes.
(Mathematica 8.01, a 36 RAM computer).
By large I am meaning 50 000 + rows, never too many columns (let's say 100 or less).
I've tried several forms to overcome the problem, to no avail. But I am sure there is an easy solution.
Can somebody out there help?
Best
Francisco
I am having trouble importing large files from Excel (or csv) into Mathematica. The program gets stuck or crashes.
(Mathematica 8.01, a 36 RAM computer).
By large I am meaning 50 000 + rows, never too many columns (let's say 100 or less).
I've tried several forms to overcome the problem, to no avail. But I am sure there is an easy solution.
Can somebody out there help?
Best
Francisco
Barrie Stokes
Dec 21, 2011, 6:55:14 AM12/21/11
to
Hi Francisco
I happened to be talking to a medical colleague about this yesterday.
Experimentation (with my 2003 version of Excel) quickly shows that the
standard Excel sheet has 256 columns (A to IV) and 2^16 rows, so 2^24
cells. I said in passing that if there were problems Importing a very
full sheet from him, I would just cut the sheet up into sufficiently
smaller subsheets (keeping all the columns, but taking rows 1-10,000,
10,001 to 20,000, or so). Then join up the subsheets in Mathematica
(trivial).
Sometimes you need to get the job done without a general exploration of
the problem - at least that*s how it often is for me.
BTW, some Googling found this at
http://msdn.microsoft.com/en-us/library/ff700514.aspx#Office2007excelPerf_BigGridIncreasedLimitsExcel:
*Starting in Excel 2007, the "Big Grid" increases the maximum number
of rows per worksheet from 65,536 to over 1 million, and the number of
columns from 256 (IV) to 16,384 (XFD).*
Best,
Barrie
>>> On 20/12/2011 at 7:01 pm, in message
<2011122008...@smc.vnet.net>,
I happened to be talking to a medical colleague about this yesterday.
Experimentation (with my 2003 version of Excel) quickly shows that the
standard Excel sheet has 256 columns (A to IV) and 2^16 rows, so 2^24
cells. I said in passing that if there were problems Importing a very
full sheet from him, I would just cut the sheet up into sufficiently
smaller subsheets (keeping all the columns, but taking rows 1-10,000,
10,001 to 20,000, or so). Then join up the subsheets in Mathematica
(trivial).
Sometimes you need to get the job done without a general exploration of
the problem - at least that*s how it often is for me.
BTW, some Googling found this at
http://msdn.microsoft.com/en-us/library/ff700514.aspx#Office2007excelPerf_BigGridIncreasedLimitsExcel:
*Starting in Excel 2007, the "Big Grid" increases the maximum number
of rows per worksheet from 65,536 to over 1 million, and the number of
columns from 256 (IV) to 16,384 (XFD).*
Best,
Barrie
>>> On 20/12/2011 at 7:01 pm, in message
<2011122008...@smc.vnet.net>,
Leonid Shifrin
Dec 21, 2011, 6:57:48 AM12/21/11
to
Hi Francisco,
I gave one possible solution to exactly this problem here:
http://stackoverflow.com/questions/7525782/import-big-files-arrays-with-mathematica/
I hope it will solve also your problem.
Regards,
Leonid
On Tue, Dec 20, 2011 at 11:01 AM, Francisco Gutierrez <
I gave one possible solution to exactly this problem here:
http://stackoverflow.com/questions/7525782/import-big-files-arrays-with-mathematica/
I hope it will solve also your problem.
Regards,
Leonid
On Tue, Dec 20, 2011 at 11:01 AM, Francisco Gutierrez <
Francisco Gutierrez
Dec 22, 2011, 4:27:17 AM12/22/11
to
Dear Leonid:
Many thanks for this, I did not reply yesterday because I had a horrible flu and did not read my mail.
The code worked fantastically. Beautiful piece of work.
However, I still have a problem. The code seems ok for a numerical database,
but I still have problems when I have a large database which contains both numeric and string variables.
Is there an adjustment for this as well?
Best regards,
Fg
Many thanks for this, I did not reply yesterday because I had a horrible flu and did not read my mail.
The code worked fantastically. Beautiful piece of work.
However, I still have a problem. The code seems ok for a numerical database,
but I still have problems when I have a large database which contains both numeric and string variables.
Is there an adjustment for this as well?
Best regards,
Fg
Leonid Shifrin
Dec 22, 2011, 4:27:47 AM12/22/11
to
Hi Francisco,
In fact, the first of the two solutions I gave in that link is supposed to
work on a general database file. I don't have time to look deeper right now, but why don't you give more details, such as a link to some sample file you're using (it better be large), and some code you tried (you can also post on StackOverflow). In this way, it will be easier to identify the problem / bottleneck.
Hope you get well!
Cheers,
Leonid
> ------------------------------
> *From:* Leonid Shifrin <lsh...@gmail.com>
> *To:* math...@smc.vnet.net
> *Sent:* Wednesday, December 21, 2011 6:55 AM
> *Subject:* Re: import issue
In fact, the first of the two solutions I gave in that link is supposed to
work on a general database file. I don't have time to look deeper right now, but why don't you give more details, such as a link to some sample file you're using (it better be large), and some code you tried (you can also post on StackOverflow). In this way, it will be easier to identify the problem / bottleneck.
Hope you get well!
Cheers,
Leonid
> *From:* Leonid Shifrin <lsh...@gmail.com>
> *To:* math...@smc.vnet.net
> *Sent:* Wednesday, December 21, 2011 6:55 AM
> *Subject:* Re: import issue
0 new messages