in-place concatenation?

[Seth P]

I'm doing something like this to add a new DataFrame to an existing one:
    df1 = pd.concat((df1, df2))
The problem is, I believe, that this temporarily doubles memory usage, which can be problematic when df1 is very large.
Is there a way to do in-place concatenation? (In case it matters/helps, df1 and df2 have the same columns and non-overlapping index.) Alas DataFrame.merge() doesn't seem to have an in_place option.

Of course I also want to do in-place concatenation of Panel objects...

[Jeff Reback]

best way is to append both to a HDFStore (could be in chunks if they r really large)

then read in the store

works for panels as well

cannot so inplace concat easily

in theory u could resize a numpy array and copy the data to it 

but I don't think that saves anything

--
You received this message because you are subscribed to the Google Groups "PyData" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydata+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Stephan Hoyer]

Unfortunately, concatenating a dataframe in place is not possible, nor is it possible to "resize" a numpy array, due to the numpy memory model -- an array is a contiguous block of memory:

http://docs.scipy.org/doc/numpy/reference/internals.html

I agree with Jeff that the best solution (if you have memory concerns) is probably to append to an HDF5 table, which does not need to copy to append:

http://pandas.pydata.org/pandas-docs/stable/io.html#table-format

You could do the same thing with a database, which can also append rows efficiently.

[Seth P]

FWIW, the documentation for DataFrame.update(), http://pandas.pydata.org/pandas-docs/dev/generated/pandas.DataFrame.update.html#pandas.DataFrame.update, says that it supports "join : {‘left’, ‘right’, ‘outer’, ‘inner’}, default ‘left’", but in practice it complains if join is anything other than 'left'.

Also, there seems to be an inconsistency in naming the {‘left’, ‘right’, ‘outer’, ‘inner’} parameter: DataFrame.update() uses "join"; while DataFrame.join() and DataFrame.merge() uses "how".