I am processing large files of numerical data.  Each line is either a
single (positive) integer, or a pair of positive integers, where the
second represents the number of times that the first number is
repeated in the data -- this is to avoid generating huge raw files,
since one particular number is often repeated in the data generation
step.

My question is how to process such files efficiently to obtain a
frequency histogram of the data (how many times each number occurs in
the data, taking into account the repetitions).  My current code is as
follows:

-------------------
#!/usr/bin/env python
# Counts the occurrences of integers in a file and makes a histogram
of them
# Allows for a second field which gives the number of counts of each
datum

import sys
args = sys.argv
num_args = len(args)

if num_args < 2:
        print "Syntaxis: count.py archivo"
        sys.exit();

name = args[1]
file = open(name, "r")

hist = {}   # dictionary for histogram
num = 0

for line in file:
        data = line.split()
        first = int(data[0])

        if len(data) == 1:
                count = 1
        else:
                count = int(data[1])    # more than one repetition

        if first in hist:       # add the information to the histogram
                hist[first]+=count
        else:
                hist[first]=count

        num+=count

keys = hist.keys()
keys.sort()

print "# i  fraction   hist[i]"
for i in keys:
        print i, float(hist[i])/num, hist[i]
---------------------

The data files are large (~100 million lines), and this code takes a
long time to run (compared to just doing wc -l, for example).

Am I doing something very inefficient?  (Any general comments on my
pythonic (or otherwise) style are also appreciated!)  Is
"line.split()" efficient, for example?

Is a dictionary the right way to do this?  In any given file, there is
an upper bound on the data, so it seems to me that some kind of array
(numpy?) would be more efficient, but the upper bound changes in each
file.

- How often does a single number occur compared to a pair of numbers ?
E.g. if a single number is much more common that a pair, you can avoid
split() most of the times:
    try: first,count = int(line), 1
    except ValueError:
        first,count = map(int,line.split())

Similarly if the pair is much more frequent than the single number,
just invert the above so that the common case is in the 'try' block
and the infrequent in 'except'. However if the two cases have similar
frequency, or if you have no a priori knowledge, try/except will
likely be slower.

- What proportion of the first numbers is unique ? If it's small
enough, a faster way to update the dict is:
        try: hist[first]+=count
        except KeyError:
                hist[first] = 1

from collections import defaultdict

def get_hist(file_name):
    hist = defaultdict(int)
    f = open(filename,"r")
    for line in f:
        vals = line.split()
        val = int(vals[0])
        try: # don't look to see if you will cause an error,
             # just cause it and then deal with it
            cnt = int(vals[1])
        except IndexError:
            cnt = 1
        hist[val] += cnt
    return hist

HTH

Keep an eye on your program's memory consumption as it runs.  The
overhead of a pair of python ints and a dictionary cell to hold them
is some dozens of bytes at minimum.  If you have a lot of distinct
keys and not enough memory to hold them all in the large dict, your
system may be thrashing.  If that is happening, the two basic
solutions are 1) buy more memory; or, 2) divide the input into smaller
pieces, attack them separately, and merge the results.

  for line in file:
    pass

to see how much time is consumed merely by iterating over the
file.  This should give you a baseline from which you can base
your timings

  try:
    first, count = map(int, data)
  except:
    first = int(data[0])
    count = 1

or possibly

  first = int(data[0])
  try:
    count = int(data[1])
  except:
    count = 0

or even

  # pad it to contain at least two items
  # then slice off the first two
  # and then map() calls to int()
  first, count = map(int,(data + [1])[:2])

I don't know how efficient len() is (if it's internally linearly
counting the items in data, or if it's caching the length as data
is created/assigned/modifed) and how that efficiency compares to
try/except blocks, map() or int() calls.

I'm not sure any of them is more or less "pythonic", but they
should all do the same thing.

  hist[first] = hist.get(first, 0) + count

  hist[first] += count

and then you'd iterate over hist (which would already be sorted
because it's in index order) and use those where count != 0 to
avoid the holes.

Otherwise, your code looks good...the above just riff on various
ways of rewriting your code in case one nets you extra
time-savings per loop.

Lists, tuples and dicts (and maybe strings?) know how long they are, so
len() takes essentially constant time.

Custom classes are free to define __len__ anyway they like.

class MyList(list):
    """Just like the built-in list, but stupider."""
    def __len__(self):
        # Untested, for obvious reasons.
        import random
        import sys
        while True:
            guess = random.randrange(0, sys.maxint)
            count = 0
            for i in self:
                count += 1
            if count == guess:
                return count

Caching __len__ is left as an exercise to the reader...

("Why does it take so long to make a three-tier wedding cake? I can boil
an egg in three minutes!!!")

import psyco
from collections import defaultdict

def get_hist(file_name):
    hist = defaultdict(int)

    for line in open(file_name):
        if " " in line:
            pair = line.split()
            hist[int(pair[0])] += int(pair[1])
        else:
            hist[int(line)] += 1

    return hist

psyco.bind(get_hist)

It doesn't work if lines may contain spurious spaces...

import std.stdio, std.conv, std.string, std.stream;

int[int] get_hist(string file_name) {
    int[int] hist;

    foreach(string line; new BufferedFile(file_name)) {
        int pos = find(line, ' ');
        if (pos == -1)
            hist[toInt(line)]++;
        else
            hist[toInt(line[0 .. pos])] += toInt(line[pos+1 .. $]);
    }

    return hist;

Maybe if you didn't have a homegrown file format, there would have
been readymade histogram utilities?  Or at least a good reason to
spend the time writing an optimized C version.

/Jorgen