Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Editing large text (*.CSV) files
61 views
Skip to first unread message
ShotSimon
Aug 11, 2008, 1:10:12 PM8/11/08
to
I inadvertently created a 3.5Gb file with data using the Write Characters to File.vi and giving it a *.csv extension. I have seen some info on the forums dealing with files this large but noticed that they point to Open G for the most part. I was wondering if someone knew how to quickly divide this file up possibly using the DOS "TYPE" command with echo turned off? This seems to open the file but echoing to the screen takes a really long time. I tried downloading other text editors with not much luck, EditPad Pro 6 (2GB), and UltraEdit which I forced closed after 10mins.
The wild thing is LabVIEW let me create this large file but an't edit it:smileyvery-happy:
The data format is as follows:
1, 21, 2, 9, 16.8652802, 21, 2, 10, 16.834560.
.
.
Regards,
-SS
Message Edited by ShotSimon on 08-11-2008 12:09 PM
The wild thing is LabVIEW let me create this large file but an't edit it:smileyvery-happy:
The data format is as follows:
1, 21, 2, 9, 16.8652802, 21, 2, 10, 16.834560.
.
.
Regards,
-SS
Message Edited by ShotSimon on 08-11-2008 12:09 PM
Brad Turpin
Aug 12, 2008, 4:40:28 PM8/12/08
to
Hi SS,
My recommendation is to immediately convert that 3.5 GB ASCII file into a binary file. You could do this in LabVIEW by reading 1000 lines at a time in a loop and appending the parsed block of values to the end of a TDMS file with the TDMS Write.vi. You probably won't be able to load all the values from that file into RAM, so that's why I suggest you load the data buffer by buffer fromt he ASCII file and convert each data block from ASCII into binary. A binary file should be smaller and will load MUCH faster into all possible environments (LV, DIAdem, Excel), but still you probably won't be able to load the whole data file into RAM. So you'll need to either process/graph the data in batches, or you'll need to condense each set of N values into one representative value (average) or perhaps two representative values (min and max). This is what DIAdem calls loading with data reduction, and it effectively results in a downsampling of your data set to reduce its size. If you acquired your data 5 times faster than you strictly speaking needed to, say, then condensing N values into 1 would result in no loss of information but would enable LV or DIAdem to load the roughly 700 MB of resulting downsampled data into RAM. You could even do this in your ASCII to binary conversion loop to create a much smaller binary file right away.
DIAdem offers serveral methods for dealing with large data sets, but no application is going to deal well with 3.5 GB of ASCII data. Let us know if you're interested in exploring any fo the DIAdem options.
Brad TurpinDIAdem Product Support EngineerNational Instruemnts
My recommendation is to immediately convert that 3.5 GB ASCII file into a binary file. You could do this in LabVIEW by reading 1000 lines at a time in a loop and appending the parsed block of values to the end of a TDMS file with the TDMS Write.vi. You probably won't be able to load all the values from that file into RAM, so that's why I suggest you load the data buffer by buffer fromt he ASCII file and convert each data block from ASCII into binary. A binary file should be smaller and will load MUCH faster into all possible environments (LV, DIAdem, Excel), but still you probably won't be able to load the whole data file into RAM. So you'll need to either process/graph the data in batches, or you'll need to condense each set of N values into one representative value (average) or perhaps two representative values (min and max). This is what DIAdem calls loading with data reduction, and it effectively results in a downsampling of your data set to reduce its size. If you acquired your data 5 times faster than you strictly speaking needed to, say, then condensing N values into 1 would result in no loss of information but would enable LV or DIAdem to load the roughly 700 MB of resulting downsampled data into RAM. You could even do this in your ASCII to binary conversion loop to create a much smaller binary file right away.
DIAdem offers serveral methods for dealing with large data sets, but no application is going to deal well with 3.5 GB of ASCII data. Let us know if you're interested in exploring any fo the DIAdem options.
Brad TurpinDIAdem Product Support EngineerNational Instruemnts
ShotSimon
Aug 12, 2008, 5:10:12 PM8/12/08
to
Brad,
Thanks for the reply, sorry I accidentally posted to the DIAdem forum instead of LabVIEW. Turns out that LabVIEW can create a 3.5GB file just fine. Also I was able to break it down into 1MB chunks using code <a href="http://forums.ni.com/ni/board/message?board.id=170&message.id=348433#M348433" target="_blank">from this reposted thread</a>. So the short answer is it's probably not good practice but LabVIEW can do it depending on your system resources and which file handlers you use.
Regards,
-SS
<a href="http://forums.ni.com/ni/board/message?board.id=170&message.id=348433#M348433" target="_blank"></a>
Thanks for the reply, sorry I accidentally posted to the DIAdem forum instead of LabVIEW. Turns out that LabVIEW can create a 3.5GB file just fine. Also I was able to break it down into 1MB chunks using code <a href="http://forums.ni.com/ni/board/message?board.id=170&message.id=348433#M348433" target="_blank">from this reposted thread</a>. So the short answer is it's probably not good practice but LabVIEW can do it depending on your system resources and which file handlers you use.
Regards,
-SS
<a href="http://forums.ni.com/ni/board/message?board.id=170&message.id=348433#M348433" target="_blank"></a>
0 new messages