Newsgroups: comp.lang.python
From: David Sanders <dpsand...@gmail.com>
Date: Fri, 18 Jan 2008 09:15:58 -0800 (PST)
Local: Fri, Jan 18 2008 12:15 pm
Subject: Efficient processing of large nuumeric data file
Hi,
I am processing large files of numerical data. Each line is either a My question is how to process such files efficiently to obtain a ------------------- import sys if num_args < 2: name = args[1] hist = {} # dictionary for histogram for line in file: if len(data) == 1: if first in hist: # add the information to the histogram num+=count keys = hist.keys() print "# i fraction hist[i]" The data files are large (~100 million lines), and this code takes a Am I doing something very inefficient? (Any general comments on my Is a dictionary the right way to do this? In any given file, there is Thanks and best wishes, You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
| ||||||||||||||